2024-02-20T00:12:18.672 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-20T00:12:18.672 DEBUG:teuthology.run:Teuthology command: teuthology --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi --archive /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154 --owner scheduled_gabrioux@teuthology -- /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/orig.config.yaml 2024-02-20T00:12:18.720 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T00:12:18.817 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154 branch: wip-guits-testing-6-2024-02-19-2119-pacific description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: gabrioux@ibm.com first_in_suite: false job_id: '7568154' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-guits-testing-6-2024-02-19-2119-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-guits-testing-6-2024-02-19-2119-pacific sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5185 sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd sleep_before_teardown: 0 subset: 1/999 suite: orch:cephadm suite_branch: wip-guits-testing-6-2024-02-19-2119-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ0W8H30qUQ4q7DBgNdqzwBG8unR5I58XXYMyRLBdfufnsdKVN1keaU5CaMPJRrj8msTB3Ic+b80e1pw2F70iHM= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7d1fye6oSJY3TGwuzE76VGS02yhrXNR5+sAcvx8Uv1YiDmq/AzXKfmI+CvEyl5pFfKeeEClt8RljHOOWxfu94= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_22:54:55 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-20T00:12:18.817 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa; will attempt to use it 2024-02-20T00:12:18.818 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks 2024-02-20T00:12:18.818 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-20T00:12:18.820 INFO:teuthology.task.internal:Checking packages... 2024-02-20T00:12:18.843 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19854089e18d4f65dda2b6cd74e737367c2514bd' 2024-02-20T00:12:18.843 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T00:12:18.843 INFO:teuthology.packaging:ref: None 2024-02-20T00:12:18.843 INFO:teuthology.packaging:tag: None 2024-02-20T00:12:18.843 INFO:teuthology.packaging:branch: wip-guits-testing-6-2024-02-19-2119-pacific 2024-02-20T00:12:18.843 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:12:18.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-guits-testing-6-2024-02-19-2119-pacific 2024-02-20T00:12:18.988 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-549.g19854089 2024-02-20T00:12:18.990 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-20T00:12:18.996 INFO:teuthology.task.internal:no buildpackages task found 2024-02-20T00:12:18.997 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-20T00:12:19.003 INFO:teuthology.task.internal:Saving configuration 2024-02-20T00:12:19.020 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-20T00:12:19.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T00:12:19.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 00:07:10.550577', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ0W8H30qUQ4q7DBgNdqzwBG8unR5I58XXYMyRLBdfufnsdKVN1keaU5CaMPJRrj8msTB3Ic+b80e1pw2F70iHM='} 2024-02-20T00:12:19.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 00:07:10.548640', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7d1fye6oSJY3TGwuzE76VGS02yhrXNR5+sAcvx8Uv1YiDmq/AzXKfmI+CvEyl5pFfKeeEClt8RljHOOWxfu94='} 2024-02-20T00:12:19.077 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-20T00:12:19.083 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-20T00:12:19.083 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-20T00:12:19.083 INFO:teuthology.run_tasks:Running task console_log... 2024-02-20T00:12:19.158 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a4f677f70>, signals=[15]) 2024-02-20T00:12:19.158 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-20T00:12:19.164 INFO:teuthology.task.internal:Opening connections... 2024-02-20T00:12:19.165 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-02-20T00:12:19.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:12:19.242 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-02-20T00:12:19.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:12:19.315 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-20T00:12:19.322 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-02-20T00:12:19.344 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-02-20T00:12:19.344 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8" 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2024-02-20T00:12:19.404 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T00:12:19.405 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T00:12:19.405 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-02-20T00:12:19.427 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-02-20T00:12:19.449 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-02-20T00:12:19.450 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-20T00:12:19.511 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-02-20T00:12:19.512 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T00:12:19.512 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-20T00:12:19.512 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T00:12:19.512 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-02-20T00:12:19.536 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-20T00:12:19.545 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-20T00:12:19.551 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-20T00:12:19.551 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-02-20T00:12:19.553 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-02-20T00:12:19.573 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-20T00:12:19.579 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-20T00:12:19.579 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-02-20T00:12:19.613 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-02-20T00:12:19.646 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-20T00:12:19.703 INFO:teuthology.run_tasks:Running task kernel... 2024-02-20T00:12:19.717 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-20T00:12:19.717 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-20T00:12:19.717 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T00:12:19.718 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T00:12:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:12:19.737 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-02-20T00:12:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:12:19.738 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-20T00:12:19.796 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-489.el8.x86_64 2024-02-20T00:12:19.796 INFO:teuthology.task.kernel:Running kernel on smithi087: 4.18.0-489.el8.x86_64 2024-02-20T00:12:19.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-02-20T00:12:19.798 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-489.el8.x86_64 2024-02-20T00:12:19.798 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-489.el8.x86_64 2024-02-20T00:12:19.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-02-20T00:12:28.269 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 4.6 MB/s | 27 MB 00:05 2024-02-20T00:12:28.451 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 27 MB 00:06 2024-02-20T00:12:34.236 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 9.0 MB/s | 10 MB 00:01 2024-02-20T00:12:34.376 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 9.2 MB/s | 10 MB 00:01 2024-02-20T00:12:35.978 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 155 kB/s | 18 kB 00:00 2024-02-20T00:12:36.126 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 123 kB/s | 18 kB 00:00 2024-02-20T00:12:36.296 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2024-02-20T00:12:36.427 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 36 kB/s | 6.9 kB 00:00 2024-02-20T00:12:36.556 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-02-20T00:12:36.670 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-20T00:12:37.615 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-02-20T00:12:41.390 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 300 kB/s | 24 kB 00:00 2024-02-20T00:12:42.745 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 16 MB 00:05 2024-02-20T00:12:44.320 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T00:12:44.320 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T00:12:44.431 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:12:44.431 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T00:12:44.432 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T00:12:44.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:12:44.433 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:12:44.433 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:12:44.433 INFO:teuthology.orchestra.run.smithi087.stdout:Install 3 Packages 2024-02-20T00:12:44.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:12:44.434 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 90 M 2024-02-20T00:12:44.434 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 97 M 2024-02-20T00:12:44.434 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:12:45.583 INFO:teuthology.orchestra.run.smithi087.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 9.3 MB/s | 10 MB 00:01 2024-02-20T00:12:46.483 INFO:teuthology.orchestra.run.smithi087.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-02-20T00:12:46.487 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 448 kB/s | 24 kB 00:00 2024-02-20T00:12:47.042 INFO:teuthology.orchestra.run.smithi087.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-02-20T00:12:47.042 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:12:47.042 INFO:teuthology.orchestra.run.smithi087.stdout:Total 34 MB/s | 90 MB 00:02 2024-02-20T00:12:47.722 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:12:48.140 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:12:48.140 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:12:49.382 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:12:49.408 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:12:49.421 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-20T00:12:49.422 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-20T00:12:49.541 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-20T00:12:49.542 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout:Install 3 Packages 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:12:49.543 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 90 M 2024-02-20T00:12:49.544 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 97 M 2024-02-20T00:12:49.544 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:12:51.163 INFO:teuthology.orchestra.run.smithi195.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.5 MB/s | 10 MB 00:01 2024-02-20T00:12:52.124 INFO:teuthology.orchestra.run.smithi195.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-20T00:12:52.162 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:12:52.623 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:12:52.889 INFO:teuthology.orchestra.run.smithi195.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-02-20T00:12:52.889 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:12:52.889 INFO:teuthology.orchestra.run.smithi195.stdout:Total 27 MB/s | 90 MB 00:03 2024-02-20T00:12:53.572 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:12:53.990 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:12:53.990 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:12:55.225 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:12:55.260 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:12:56.179 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:12:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:12:58.052 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:12:58.564 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:13:00.021 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:13:00.312 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:02.111 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:13:02.471 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:13:05.909 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:13:06.221 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:22.269 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:22.294 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:22.659 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:22.659 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:13:22.659 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:13:23.166 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:23.166 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:13:23.166 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:13:23.166 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T00:13:23.167 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T00:13:23.167 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:13:23.167 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:13:23.335 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T00:13:23.556 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T00:13:23.556 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T00:13:23.556 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T00:13:23.557 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T00:13:23.557 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T00:13:23.557 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-20T00:13:23.557 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T00:13:23.557 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T00:13:23.767 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T00:13:23.767 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T00:13:23.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-02-20T00:13:23.800 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-02-20T00:13:23.800 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-02-20T00:13:23.800 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.02 2024-02-20T00:13:23.800 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 129.el8 2024-02-20T00:13:23.800 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Group : System Environment/Base 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 9265586 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Relocations : (not relocatable) 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : CentOS BuildSystem 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-02-20T00:13:23.801 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:13:23.802 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T00:13:23.803 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T00:13:23.803 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 4.18.0-539.el8.x86_64 2024-02-20T00:13:23.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T00:13:25.173 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-02-20T00:13:27.100 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-02-20T00:13:27.102 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-02-20T00:13:27.138 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.THJDppIA9K 2024-02-20T00:13:27.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.THJDppIA9K 2024-02-20T00:13:27.222 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.THJDppIA9K 2024-02-20T00:13:27.327 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.THJDppIA9K is 6KB 2024-02-20T00:13:27.376 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.THJDppIA9K 2024-02-20T00:13:27.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T00:13:27.459 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T00:13:27.459 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T00:13:27.459 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T00:13:27.459 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T00:13:27.460 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T00:13:27.620 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-02-20T00:13:28.071 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:28.095 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:28.452 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:28.453 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-20T00:13:28.453 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:13:28.876 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:13:29.545 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T00:13:29.764 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T00:13:29.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T00:13:29.765 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-20T00:13:29.765 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T00:13:29.765 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T00:13:29.765 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-20T00:13:29.765 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T00:13:29.765 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T00:13:29.977 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-20T00:13:29.978 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-20T00:13:29.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.02 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 129.el8 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Group : System Environment/Base 2024-02-20T00:13:30.011 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 9265586 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Relocations : (not relocatable) 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : CentOS BuildSystem 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T00:13:30.012 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T00:13:30.013 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T00:13:30.013 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-02-20T00:13:30.013 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:13:30.013 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T00:13:30.013 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T00:13:30.014 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 4.18.0-539.el8.x86_64 2024-02-20T00:13:30.014 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T00:13:31.358 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-02-20T00:13:33.315 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-02-20T00:13:33.316 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-02-20T00:13:33.341 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.t7faJqMjB5 2024-02-20T00:13:33.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.t7faJqMjB5 2024-02-20T00:13:33.415 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.t7faJqMjB5 2024-02-20T00:13:33.558 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.t7faJqMjB5 is 6KB 2024-02-20T00:13:33.606 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.t7faJqMjB5 2024-02-20T00:13:33.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T00:13:33.697 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-20T00:13:33.698 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-20T00:13:33.698 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-20T00:13:33.698 INFO:teuthology.orchestra.run.smithi195.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-20T00:13:33.699 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-20T00:13:33.857 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-02-20T00:13:57.646 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-20T00:13:57.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-20T00:13:57.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:14:03.865 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-20T00:14:03.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-20T00:14:03.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:14:57.652 DEBUG:teuthology.orchestra.remote:timed out 2024-02-20T00:15:03.872 DEBUG:teuthology.orchestra.remote:timed out 2024-02-20T00:15:06.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-20T00:15:06.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:15:06.933 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-02-20T00:15:07.297 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-20T00:15:07.297 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-20T00:15:07.297 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-02-20T00:15:07.314 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-539.el8.x86_64 2024-02-20T00:15:07.314 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T00:15:07.314 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T00:15:07.314 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T00:15:08.315 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-20T00:15:08.316 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T00:15:08.402 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-02-20T00:15:08.417 DEBUG:teuthology.parallel:result is None 2024-02-20T00:15:12.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-20T00:15:12.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:15:13.191 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-20T00:15:13.480 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-20T00:15:13.480 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-20T00:15:13.481 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-02-20T00:15:13.497 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-539.el8.x86_64 2024-02-20T00:15:13.497 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-20T00:15:13.497 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T00:15:13.498 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T00:15:14.498 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-20T00:15:14.498 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T00:15:14.550 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-02-20T00:15:14.568 DEBUG:teuthology.parallel:result is None 2024-02-20T00:15:14.568 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-20T00:15:14.576 INFO:teuthology.task.internal:Creating test directory... 2024-02-20T00:15:14.576 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T00:15:14.579 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T00:15:14.600 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-20T00:15:14.606 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-20T00:15:14.613 INFO:teuthology.task.internal:Creating archive directory... 2024-02-20T00:15:14.613 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T00:15:14.643 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T00:15:14.672 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-20T00:15:14.678 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-20T00:15:14.679 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T00:15:14.710 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T00:15:14.741 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T00:15:14.752 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T00:15:14.774 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T00:15:14.784 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T00:15:14.785 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-20T00:15:14.793 INFO:teuthology.task.internal:Configuring sudo... 2024-02-20T00:15:14.793 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T00:15:14.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T00:15:14.824 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-20T00:15:14.836 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-20T00:15:14.836 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T00:15:14.866 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T00:15:14.882 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T00:15:14.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T00:15:15.025 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T00:15:15.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T00:15:15.153 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:15:15.153 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T00:15:15.219 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T00:15:15.264 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T00:15:15.315 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T00:15:15.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T00:15:15.450 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:15:15.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T00:15:15.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-02-20T00:15:15.523 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-02-20T00:15:15.592 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:15:15.624 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:15:16.058 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-20T00:15:16.066 INFO:teuthology.task.internal:Starting timer... 2024-02-20T00:15:16.066 INFO:teuthology.run_tasks:Running task pcp... 2024-02-20T00:15:16.076 INFO:teuthology.run_tasks:Running task selinux... 2024-02-20T00:15:16.084 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-20T00:15:16.085 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-02-20T00:15:16.200 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-02-20T00:15:16.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-02-20T00:15:16.283 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-02-20T00:15:16.285 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-20T00:15:16.285 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-02-20T00:15:16.309 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-02-20T00:15:16.309 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-02-20T00:15:16.339 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-02-20T00:15:16.339 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-02-20T00:15:16.339 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:15:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:15:16.377 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:15:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:15:16.410 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-20T00:15:16.410 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-02-20T00:15:16.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-02-20T00:15:16.499 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-20T00:15:16.511 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-20T00:15:16.518 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-20T00:15:16.645 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-20T00:15:16.662 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-20T00:15:16.663 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T00:21:00.997 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-02-20T00:21:00.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-20T00:21:01.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:21:01.078 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-02-20T00:21:01.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-20T00:21:01.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-20T00:21:01.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:21:01.221 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-02-20T00:21:01.289 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-02-20T00:21:01.289 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T00:21:01.299 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T00:21:01.300 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T00:21:01.300 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:21:01.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T00:21:01.302 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:21:01.332 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T00:21:01.346 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T00:21:01.360 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T00:21:01.374 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T00:21:01.410 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-02-20T00:21:01.422 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-02-20T00:21:01.436 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T00:21:01.442 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-02-20T00:21:01.449 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T00:21:01.453 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-02-20T00:21:01.467 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T00:21:01.480 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T00:21:01.517 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2024-02-20T00:21:01.520 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:21:01.520 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-02-20T00:21:01.521 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.521 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.521 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.521 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.549 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.553 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:21:01.554 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T00:21:01.563 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T00:21:01.564 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi087.front.sepia.ceph.com 2024-02-20T00:21:01.564 DEBUG:teuthology.orchestra.run.smithi087:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:21:01.564 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2024-02-20T00:21:01.564 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:21:02.339 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:22 ago on Tue 20 Feb 2024 12:19:40 AM UTC. 2024-02-20T00:21:02.380 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:22 ago on Tue 20 Feb 2024 12:19:40 AM UTC. 2024-02-20T00:21:04.376 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:21:04.377 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:04.377 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:21:04.377 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:04.377 INFO:teuthology.orchestra.run.smithi087.stdout:Resetting modules: 2024-02-20T00:21:04.377 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools 2024-02-20T00:21:04.378 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:04.378 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:21:04.378 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:04.378 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:04.392 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout:Resetting modules: 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:04.394 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:04.633 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:21:04.642 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:21:05.464 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:25 ago on Tue 20 Feb 2024 12:19:40 AM UTC. 2024-02-20T00:21:05.487 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:25 ago on Tue 20 Feb 2024 12:19:40 AM UTC. 2024-02-20T00:21:07.475 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout:Installing group/module packages: 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T00:21:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout:Installing module profiles: 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools/common 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout:Enabling module streams: 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools rhel8 2024-02-20T00:21:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 28 M 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 92 M 2024-02-20T00:21:07.513 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:21:07.517 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout:Installing group/module packages: 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T00:21:07.553 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout:Installing module profiles: 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools/common 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout:Enabling module streams: 2024-02-20T00:21:07.554 INFO:teuthology.orchestra.run.smithi195.stdout: container-tools rhel8 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 28 M 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 92 M 2024-02-20T00:21:07.555 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:21:07.740 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 4.7 MB/s | 688 kB 00:00 2024-02-20T00:21:07.774 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 5.7 MB/s | 1.0 MB 00:00 2024-02-20T00:21:07.799 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2024-02-20T00:21:07.824 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 5.0 MB/s | 257 kB 00:00 2024-02-20T00:21:07.835 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 5.1 MB/s | 688 kB 00:00 2024-02-20T00:21:07.866 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 4.2 MB/s | 180 kB 00:00 2024-02-20T00:21:07.885 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 757 kB/s | 38 kB 00:00 2024-02-20T00:21:07.908 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 595 kB/s | 25 kB 00:00 2024-02-20T00:21:07.943 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): crun-1.14-1.module_el8+871+552be958.x86 4.3 MB/s | 257 kB 00:00 2024-02-20T00:21:07.958 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-02-20T00:21:07.977 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.7 MB/s | 1.0 MB 00:00 2024-02-20T00:21:08.061 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.1 MB/s | 180 kB 00:00 2024-02-20T00:21:08.075 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 16 MB/s | 4.3 MB 00:00 2024-02-20T00:21:08.111 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 496 kB/s | 25 kB 00:00 2024-02-20T00:21:08.187 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): netavark-1.10.2-1.module_el8+885+7da147 18 MB/s | 4.3 MB 00:00 2024-02-20T00:21:08.212 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 937 kB/s | 94 kB 00:00 2024-02-20T00:21:08.276 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 14 MB/s | 9.7 MB 00:00 2024-02-20T00:21:08.320 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 16 MB/s | 9.7 MB 00:00 2024-02-20T00:21:08.335 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 9.8 MB/s | 2.5 MB 00:00 2024-02-20T00:21:08.360 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 629 kB/s | 53 kB 00:00 2024-02-20T00:21:08.446 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 421 kB/s | 53 kB 00:00 2024-02-20T00:21:08.519 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 16 MB/s | 8.8 MB 00:00 2024-02-20T00:21:08.543 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): yajl-2.1.0-12.el8.x86_64.rpm 196 kB/s | 41 kB 00:00 2024-02-20T00:21:08.544 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:21:08.544 INFO:teuthology.orchestra.run.smithi195.stdout:Total 28 MB/s | 28 MB 00:00 2024-02-20T00:21:08.562 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): skopeo-1.14.3-0.1.module_el8+885+7da14 23 MB/s | 8.8 MB 00:00 2024-02-20T00:21:08.613 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 245 kB/s | 41 kB 00:00 2024-02-20T00:21:08.663 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): toolbox-0.0.99.5-1.module_el8+847+7863 5.6 MB/s | 2.5 MB 00:00 2024-02-20T00:21:08.663 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:21:08.663 INFO:teuthology.orchestra.run.smithi087.stdout:Total 24 MB/s | 28 MB 00:01 2024-02-20T00:21:08.815 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:21:08.836 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:21:08.836 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:21:08.932 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:21:08.952 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:21:08.952 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:21:09.099 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:21:09.102 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:21:09.212 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:21:09.216 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:21:09.790 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:21:09.923 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:21:10.727 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T00:21:10.869 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T00:21:10.946 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:21:11.081 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:21:11.124 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T00:21:11.243 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T00:21:11.303 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T00:21:11.453 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T00:21:11.527 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T00:21:11.695 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T00:21:12.105 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T00:21:12.256 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T00:21:13.102 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T00:21:13.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T00:21:13.324 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T00:21:13.420 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T00:21:13.511 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T00:21:13.615 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T00:21:13.958 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T00:21:14.065 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T00:21:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:21:14.303 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:21:14.336 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:21:14.505 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:21:14.507 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:21:14.684 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T00:21:14.940 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:21:14.941 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T00:21:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T00:21:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T00:21:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:21:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T00:21:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:15.528 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:21:15.631 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T00:21:15.631 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T00:21:15.632 INFO:teuthology.orchestra.run.smithi087.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T00:21:15.633 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:15.633 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:21:15.841 DEBUG:teuthology.parallel:result is None 2024-02-20T00:21:15.894 DEBUG:teuthology.parallel:result is None 2024-02-20T00:21:15.894 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T00:21:15.904 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T00:21:15.904 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi087.front.sepia.ceph.com 2024-02-20T00:21:15.905 DEBUG:teuthology.orchestra.run.smithi087:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:21:15.905 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2024-02-20T00:21:15.905 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:21:16.118 DEBUG:teuthology.parallel:result is None 2024-02-20T00:21:16.119 DEBUG:teuthology.parallel:result is None 2024-02-20T00:21:16.119 INFO:teuthology.run_tasks:Running task install... 2024-02-20T00:21:16.128 DEBUG:teuthology.task.install:project ceph 2024-02-20T00:21:16.128 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'}} 2024-02-20T00:21:16.128 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'} 2024-02-20T00:21:16.128 INFO:teuthology.task.install:Using flavor: default 2024-02-20T00:21:16.136 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T00:21:16.136 INFO:teuthology.task.install:extra packages: [] 2024-02-20T00:21:16.136 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-20T00:21:16.136 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:21:16.136 INFO:teuthology.packaging:ref: None 2024-02-20T00:21:16.137 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:21:16.137 INFO:teuthology.packaging:branch: None 2024-02-20T00:21:16.137 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:21:16.442 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:21:16.443 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:21:16.445 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-20T00:21:16.445 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:21:16.446 INFO:teuthology.packaging:ref: None 2024-02-20T00:21:16.446 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:21:16.446 INFO:teuthology.packaging:branch: None 2024-02-20T00:21:16.446 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:21:16.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:21:16.663 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-20T00:21:16.663 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-20T00:21:16.675 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-20T00:21:16.675 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-20T00:21:16.705 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T00:21:16.706 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:21:16.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T00:21:16.714 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T00:21:16.714 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:21:16.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T00:21:16.735 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T00:21:16.736 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:21:16.736 INFO:teuthology.packaging:ref: None 2024-02-20T00:21:16.736 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:21:16.736 INFO:teuthology.packaging:branch: None 2024-02-20T00:21:16.736 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:21:16.736 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T00:21:16.743 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T00:21:16.744 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:21:16.744 INFO:teuthology.packaging:ref: None 2024-02-20T00:21:16.744 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:21:16.744 INFO:teuthology.packaging:branch: None 2024-02-20T00:21:16.744 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:21:16.744 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T00:21:16.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T00:21:16.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T00:21:16.880 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T00:21:16.889 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T00:21:16.947 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-02-20T00:21:16.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-02-20T00:21:16.955 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-02-20T00:21:16.956 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-02-20T00:21:17.211 INFO:teuthology.orchestra.run.smithi087.stdout:76 files removed 2024-02-20T00:21:17.223 INFO:teuthology.orchestra.run.smithi195.stdout:76 files removed 2024-02-20T00:21:17.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-02-20T00:21:17.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-02-20T00:21:17.728 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-02-20T00:21:17.746 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 721 kB/s | 74 kB 00:00 2024-02-20T00:21:17.846 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 243 kB/s | 15 kB 00:00 2024-02-20T00:21:17.879 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 134 kB/s | 15 kB 00:00 2024-02-20T00:21:17.947 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-20T00:21:17.979 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 25 kB/s | 1.7 kB 00:00 2024-02-20T00:21:24.113 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 27 MB 00:06 2024-02-20T00:21:24.547 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 4.1 MB/s | 27 MB 00:06 2024-02-20T00:21:30.188 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - BaseOS 9.3 MB/s | 10 MB 00:01 2024-02-20T00:21:30.628 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 8.5 MB/s | 10 MB 00:01 2024-02-20T00:21:32.239 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2024-02-20T00:21:32.698 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-02-20T00:21:32.732 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 35 kB/s | 18 kB 00:00 2024-02-20T00:21:33.049 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2024-02-20T00:21:33.400 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 8 - PowerTools 7.9 MB/s | 4.8 MB 00:00 2024-02-20T00:21:33.609 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 4.8 MB 00:00 2024-02-20T00:21:34.443 INFO:teuthology.orchestra.run.smithi195.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-02-20T00:21:34.661 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-02-20T00:21:36.230 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-20T00:21:36.968 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 16 MB 00:02 2024-02-20T00:21:40.020 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 620 kB/s | 24 kB 00:00 2024-02-20T00:21:40.751 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 397 kB/s | 24 kB 00:00 2024-02-20T00:21:43.540 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-20T00:21:43.543 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-20T00:21:43.544 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T00:21:43.545 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout:Install 25 Packages 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:21:43.546 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 59 M 2024-02-20T00:21:43.547 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:21:44.113 INFO:teuthology.orchestra.run.smithi087.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2024-02-20T00:21:44.138 INFO:teuthology.orchestra.run.smithi087.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-02-20T00:21:44.172 INFO:teuthology.orchestra.run.smithi087.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-20T00:21:44.222 INFO:teuthology.orchestra.run.smithi087.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 9.5 MB/s | 486 kB 00:00 2024-02-20T00:21:44.264 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:21:44.267 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-20T00:21:44.268 INFO:teuthology.orchestra.run.smithi195.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-20T00:21:44.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout:Install 25 Packages 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-02-20T00:21:44.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:21:44.271 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 59 M 2024-02-20T00:21:44.271 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:21:44.330 INFO:teuthology.orchestra.run.smithi087.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2024-02-20T00:21:44.355 INFO:teuthology.orchestra.run.smithi087.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-20T00:21:44.381 INFO:teuthology.orchestra.run.smithi087.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-20T00:21:44.406 INFO:teuthology.orchestra.run.smithi087.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-20T00:21:44.440 INFO:teuthology.orchestra.run.smithi087.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-20T00:21:44.473 INFO:teuthology.orchestra.run.smithi087.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-20T00:21:44.499 INFO:teuthology.orchestra.run.smithi087.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-20T00:21:44.641 INFO:teuthology.orchestra.run.smithi087.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-02-20T00:21:44.703 INFO:teuthology.orchestra.run.smithi195.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2024-02-20T00:21:44.728 INFO:teuthology.orchestra.run.smithi195.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 993 kB/s | 25 kB 00:00 2024-02-20T00:21:44.762 INFO:teuthology.orchestra.run.smithi195.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-20T00:21:44.795 INFO:teuthology.orchestra.run.smithi195.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-20T00:21:44.896 INFO:teuthology.orchestra.run.smithi195.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-20T00:21:44.921 INFO:teuthology.orchestra.run.smithi195.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-20T00:21:44.933 INFO:teuthology.orchestra.run.smithi087.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-20T00:21:44.946 INFO:teuthology.orchestra.run.smithi195.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-20T00:21:44.958 INFO:teuthology.orchestra.run.smithi087.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 316 kB/s | 145 kB 00:00 2024-02-20T00:21:44.972 INFO:teuthology.orchestra.run.smithi195.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.2 MB/s | 213 kB 00:00 2024-02-20T00:21:45.005 INFO:teuthology.orchestra.run.smithi195.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-20T00:21:45.008 INFO:teuthology.orchestra.run.smithi087.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 880 kB/s | 44 kB 00:00 2024-02-20T00:21:45.033 INFO:teuthology.orchestra.run.smithi087.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-02-20T00:21:45.039 INFO:teuthology.orchestra.run.smithi195.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-20T00:21:45.064 INFO:teuthology.orchestra.run.smithi195.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-20T00:21:45.067 INFO:teuthology.orchestra.run.smithi087.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2024-02-20T00:21:45.190 INFO:teuthology.orchestra.run.smithi195.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-02-20T00:21:45.326 INFO:teuthology.orchestra.run.smithi087.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 133 kB/s | 42 kB 00:00 2024-02-20T00:21:45.351 INFO:teuthology.orchestra.run.smithi087.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 148 kB/s | 47 kB 00:00 2024-02-20T00:21:45.435 INFO:teuthology.orchestra.run.smithi087.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 218 kB/s | 80 kB 00:00 2024-02-20T00:21:45.473 INFO:teuthology.orchestra.run.smithi195.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-02-20T00:21:45.498 INFO:teuthology.orchestra.run.smithi195.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 335 kB/s | 145 kB 00:00 2024-02-20T00:21:45.557 INFO:teuthology.orchestra.run.smithi195.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 753 kB/s | 44 kB 00:00 2024-02-20T00:21:45.583 INFO:teuthology.orchestra.run.smithi195.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-02-20T00:21:45.641 INFO:teuthology.orchestra.run.smithi195.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-02-20T00:21:45.668 INFO:teuthology.orchestra.run.smithi087.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 340 kB/s | 116 kB 00:00 2024-02-20T00:21:45.744 INFO:teuthology.orchestra.run.smithi087.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 781 kB/s | 306 kB 00:00 2024-02-20T00:21:45.768 INFO:teuthology.orchestra.run.smithi087.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 587 kB/s | 59 kB 00:00 2024-02-20T00:21:45.794 INFO:teuthology.orchestra.run.smithi087.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 504 kB/s | 181 kB 00:00 2024-02-20T00:21:45.827 INFO:teuthology.orchestra.run.smithi087.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 897 kB/s | 75 kB 00:00 2024-02-20T00:21:45.903 INFO:teuthology.orchestra.run.smithi087.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-20T00:21:45.961 INFO:teuthology.orchestra.run.smithi087.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-02-20T00:21:45.966 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:21:45.966 INFO:teuthology.orchestra.run.smithi087.stdout:Total 24 MB/s | 59 MB 00:02 2024-02-20T00:21:45.983 INFO:teuthology.orchestra.run.smithi195.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 99 kB/s | 42 kB 00:00 2024-02-20T00:21:46.032 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:21:46.033 INFO:teuthology.orchestra.run.smithi195.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 104 kB/s | 47 kB 00:00 2024-02-20T00:21:46.079 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:21:46.079 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:21:46.134 INFO:teuthology.orchestra.run.smithi195.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 163 kB/s | 80 kB 00:00 2024-02-20T00:21:46.443 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:21:46.446 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:21:46.659 INFO:teuthology.orchestra.run.smithi195.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 172 kB/s | 116 kB 00:00 2024-02-20T00:21:46.968 INFO:teuthology.orchestra.run.smithi195.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 328 kB/s | 306 kB 00:00 2024-02-20T00:21:47.152 INFO:teuthology.orchestra.run.smithi195.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 179 kB/s | 181 kB 00:01 2024-02-20T00:21:47.361 INFO:teuthology.orchestra.run.smithi195.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 84 kB/s | 59 kB 00:00 2024-02-20T00:21:47.386 INFO:teuthology.orchestra.run.smithi195.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 180 kB/s | 75 kB 00:00 2024-02-20T00:21:47.470 INFO:teuthology.orchestra.run.smithi195.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2024-02-20T00:21:47.520 INFO:teuthology.orchestra.run.smithi195.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-02-20T00:21:47.525 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:21:47.525 INFO:teuthology.orchestra.run.smithi195.stdout:Total 18 MB/s | 59 MB 00:03 2024-02-20T00:21:47.592 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:21:47.641 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:21:47.641 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:21:47.998 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:21:48.001 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:21:48.211 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:21:48.442 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-20T00:21:48.654 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-20T00:21:48.814 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:21:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:21:49.222 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:21:49.308 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-20T00:21:49.388 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:21:49.487 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:21:49.513 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-20T00:21:49.703 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:21:49.732 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:21:49.888 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:21:49.940 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:21:50.097 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:21:50.198 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-20T00:21:50.303 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:21:50.378 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:21:50.601 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:21:50.747 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-20T00:21:50.752 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:21:50.816 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:21:50.880 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:21:50.955 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:21:51.062 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-20T00:21:51.118 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-20T00:21:51.314 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:21:51.378 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:21:51.543 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:21:51.620 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-20T00:21:51.725 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:21:51.760 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:21:51.839 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:21:51.922 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:21:51.997 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:21:52.032 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-20T00:21:52.147 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:21:52.202 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:21:52.271 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:21:52.474 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:21:52.592 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:21:52.619 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:21:52.772 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:21:52.777 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:21:52.831 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:21:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:21:52.969 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:21:53.075 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-20T00:21:53.289 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-20T00:21:53.395 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:21:53.438 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:21:53.489 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:21:53.544 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:21:53.643 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:21:53.689 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-20T00:21:53.831 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-20T00:21:53.896 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-20T00:21:54.029 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-20T00:21:54.069 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-20T00:21:54.168 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:21:54.240 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:21:54.305 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-20T00:21:54.420 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-20T00:21:54.648 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-20T00:21:54.817 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-20T00:21:55.018 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-20T00:21:56.836 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:57.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:57.590 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:57.776 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:58.035 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:21:58.137 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:21:58.296 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:21:58.561 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:21:58.863 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:21:58.950 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:21:59.084 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:22:17.339 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:22:17.339 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-02-20T00:22:17.339 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-02-20T00:22:17.339 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2024-02-20T00:22:17.339 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /run 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2024-02-20T00:22:17.340 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:18.674 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-02-20T00:22:18.691 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-02-20T00:22:18.692 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-02-20T00:22:18.692 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-02-20T00:22:18.692 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2024-02-20T00:22:18.692 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:18.752 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:22:18.752 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:22:18.954 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:22:18.954 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:19.647 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:19.647 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-20T00:22:19.647 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-20T00:22:19.647 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-20T00:22:19.648 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-20T00:22:19.649 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-20T00:22:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:22:20.083 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:22:20.083 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:22:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:22:20.283 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:20.552 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-02-20T00:22:20.552 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.552 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.553 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:20.554 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:22:20.828 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-02-20T00:22:20.997 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-20T00:22:20.998 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-20T00:22:20.999 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-20T00:22:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-20T00:22:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-20T00:22:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:22:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-20T00:22:21.000 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-20T00:22:21.581 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:42 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-20T00:22:21.854 INFO:teuthology.orchestra.run.smithi195.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T00:22:21.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:21.856 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:21.856 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:22:22.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-02-20T00:22:22.923 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:42 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:22:23.125 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T00:22:23.126 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 53 M 2024-02-20T00:22:23.127 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 229 M 2024-02-20T00:22:23.128 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:22:23.392 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-02-20T00:22:23.434 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.8 MB/s | 203 kB 00:00 2024-02-20T00:22:23.484 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-02-20T00:22:23.518 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-02-20T00:22:23.543 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-02-20T00:22:24.135 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2024-02-20T00:22:24.137 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:24.137 INFO:teuthology.orchestra.run.smithi087.stdout:Total 52 MB/s | 53 MB 00:01 2024-02-20T00:22:24.180 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:22:24.201 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:22:24.201 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:22:24.474 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:24.476 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 53 M 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 229 M 2024-02-20T00:22:24.477 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:22:24.517 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:22:24.520 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:22:24.574 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 169 kB 00:00 2024-02-20T00:22:24.607 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.2 MB/s | 203 kB 00:00 2024-02-20T00:22:24.658 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-02-20T00:22:24.683 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-02-20T00:22:24.708 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-20T00:22:24.996 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:22:25.237 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T00:22:25.267 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2024-02-20T00:22:25.269 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:25.269 INFO:teuthology.orchestra.run.smithi195.stdout:Total 67 MB/s | 53 MB 00:00 2024-02-20T00:22:25.311 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:22:25.332 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:22:25.332 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:22:25.426 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T00:22:25.576 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:22:25.648 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:22:25.651 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:22:25.681 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:22:25.853 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T00:22:25.992 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:22:26.129 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:22:26.351 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T00:22:26.550 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T00:22:26.705 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:22:26.788 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:22:26.980 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T00:22:27.122 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:22:30.834 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:22:31.065 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T00:22:31.335 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:31.769 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:22:31.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-02-20T00:22:31.983 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:22:32.228 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T00:22:32.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T00:22:32.727 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:53 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:33.015 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:22:33.261 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-02-20T00:22:34.021 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:54 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:22:34.288 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:22:34.291 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:34.291 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-20T00:22:34.292 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T00:22:34.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T00:22:34.294 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout:Install 32 Packages 2024-02-20T00:22:34.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:34.296 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 M 2024-02-20T00:22:34.296 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 111 M 2024-02-20T00:22:34.296 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:22:35.078 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 127 kB/s | 6.3 kB 00:00 2024-02-20T00:22:35.128 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-20T00:22:35.186 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-02-20T00:22:35.211 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-20T00:22:35.304 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-20T00:22:35.320 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-02-20T00:22:35.362 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-20T00:22:35.408 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 373 kB/s | 39 kB 00:00 2024-02-20T00:22:35.430 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 853 kB/s | 57 kB 00:00 2024-02-20T00:22:35.571 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T00:22:35.575 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T00:22:35.576 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:35.577 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout:Install 32 Packages 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 31 M 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 111 M 2024-02-20T00:22:35.578 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:22:35.647 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 35 MB/s | 18 MB 00:00 2024-02-20T00:22:35.697 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-02-20T00:22:36.081 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 382 kB/s | 249 kB 00:00 2024-02-20T00:22:36.097 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 272 kB/s | 109 kB 00:00 2024-02-20T00:22:36.144 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 112 kB/s | 6.3 kB 00:00 2024-02-20T00:22:36.186 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-20T00:22:36.198 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 373 kB/s | 44 kB 00:00 2024-02-20T00:22:36.245 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-02-20T00:22:36.256 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-02-20T00:22:36.270 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-20T00:22:36.315 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 853 kB/s | 185 kB 00:00 2024-02-20T00:22:36.349 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2024-02-20T00:22:36.362 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-02-20T00:22:36.379 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-02-20T00:22:36.399 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.6 MB/s | 384 kB 00:00 2024-02-20T00:22:36.429 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2024-02-20T00:22:36.432 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-02-20T00:22:36.454 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 422 kB/s | 39 kB 00:00 2024-02-20T00:22:36.457 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2024-02-20T00:22:36.480 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-02-20T00:22:36.508 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2024-02-20T00:22:36.582 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 394 kB/s | 59 kB 00:00 2024-02-20T00:22:36.617 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-02-20T00:22:36.642 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2024-02-20T00:22:36.684 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 183 kB/s | 19 kB 00:00 2024-02-20T00:22:36.712 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 35 MB/s | 18 MB 00:00 2024-02-20T00:22:36.718 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-02-20T00:22:36.751 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-02-20T00:22:36.764 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2024-02-20T00:22:36.793 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-02-20T00:22:36.885 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.5 MB/s | 255 kB 00:00 2024-02-20T00:22:36.910 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-20T00:22:36.936 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 600 kB/s | 85 kB 00:00 2024-02-20T00:22:36.994 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-02-20T00:22:36.997 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:36.997 INFO:teuthology.orchestra.run.smithi087.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-20T00:22:37.076 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:22:37.116 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:22:37.116 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:22:37.131 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 382 kB/s | 249 kB 00:00 2024-02-20T00:22:37.164 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 272 kB/s | 109 kB 00:00 2024-02-20T00:22:37.223 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 475 kB/s | 44 kB 00:00 2024-02-20T00:22:37.257 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.0 MB/s | 185 kB 00:00 2024-02-20T00:22:37.290 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.5 MB/s | 173 kB 00:00 2024-02-20T00:22:37.316 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 6.4 MB/s | 384 kB 00:00 2024-02-20T00:22:37.341 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2024-02-20T00:22:37.366 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): python3-cryptography-3.2.1-7.el8.x86_6 980 kB/s | 641 kB 00:00 2024-02-20T00:22:37.417 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-02-20T00:22:37.441 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:22:37.442 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 484 kB/s | 49 kB 00:00 2024-02-20T00:22:37.444 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:22:37.467 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 585 kB/s | 59 kB 00:00 2024-02-20T00:22:37.492 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-02-20T00:22:37.518 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2024-02-20T00:22:37.543 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 478 kB/s | 24 kB 00:00 2024-02-20T00:22:37.568 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 585 kB/s | 29 kB 00:00 2024-02-20T00:22:37.593 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2024-02-20T00:22:37.619 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-02-20T00:22:37.644 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-20T00:22:37.669 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-20T00:22:37.695 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 457 kB/s | 23 kB 00:00 2024-02-20T00:22:37.720 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2024-02-20T00:22:37.722 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:37.722 INFO:teuthology.orchestra.run.smithi195.stdout:Total 14 MB/s | 31 MB 00:02 2024-02-20T00:22:37.803 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:22:37.841 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:22:37.842 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:22:38.161 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:22:38.164 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:22:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:22:38.771 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T00:22:38.982 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T00:22:39.215 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T00:22:39.268 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:22:39.397 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T00:22:39.458 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T00:22:39.578 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T00:22:39.677 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T00:22:39.749 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T00:22:39.901 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T00:22:39.910 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T00:22:40.087 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T00:22:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T00:22:40.270 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T00:22:40.274 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T00:22:40.446 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T00:22:40.475 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T00:22:40.598 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T00:22:40.705 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T00:22:40.759 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T00:22:40.935 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T00:22:40.945 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T00:22:41.131 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T00:22:41.296 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T00:22:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T00:22:41.518 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T00:22:41.601 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T00:22:41.664 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T00:22:41.915 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T00:22:41.994 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T00:22:42.066 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:22:42.255 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T00:22:42.323 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T00:22:42.444 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T00:22:42.529 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T00:22:42.685 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T00:22:42.700 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T00:22:42.845 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:22:42.853 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T00:22:43.111 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T00:22:43.161 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T00:22:43.310 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T00:22:43.349 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T00:22:43.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T00:22:43.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T00:22:43.716 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T00:22:43.879 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T00:22:43.974 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T00:22:44.172 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T00:22:44.180 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T00:22:44.406 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T00:22:44.542 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-20T00:22:44.739 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T00:22:44.779 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:22:45.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T00:22:45.436 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-20T00:22:45.648 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:22:46.307 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:22:46.554 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:22:46.988 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:22:47.180 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:22:47.195 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:22:47.418 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:22:47.422 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:22:47.615 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:22:47.632 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:22:47.747 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:22:47.857 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:22:48.114 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T00:22:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T00:22:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T00:22:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T00:22:48.336 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:22:48.552 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:22:48.569 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:22:48.718 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T00:22:48.952 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:48.953 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-20T00:22:49.113 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T00:22:49.114 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T00:22:49.115 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T00:22:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T00:22:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T00:22:49.116 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T00:22:49.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:49.946 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T00:22:49.947 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:49.948 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:22:49.993 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:10 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:22:50.226 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-02-20T00:22:50.989 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:10 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:22:51.506 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:22:51.554 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:22:51.555 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:22:51.555 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:22:51.651 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-02-20T00:22:52.398 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:13 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:22:52.480 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:22:52.529 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:22:52.530 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:22:52.530 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:22:52.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-02-20T00:22:53.375 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:13 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:22:53.951 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:53.952 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:22:53.953 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:53.953 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 71 k 2024-02-20T00:22:53.953 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 290 k 2024-02-20T00:22:53.953 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:22:53.997 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2024-02-20T00:22:53.998 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:53.998 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-20T00:22:53.998 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:22:54.005 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:22:54.005 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:22:54.007 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:22:54.010 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:22:54.052 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:22:54.566 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-20T00:22:54.934 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 71 k 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 290 k 2024-02-20T00:22:54.935 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:22:54.942 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:54.968 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 71 kB 00:00 2024-02-20T00:22:54.968 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:54.969 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.1 MB/s | 71 kB 00:00 2024-02-20T00:22:54.969 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:22:54.976 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:22:54.976 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:22:54.977 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:22:54.980 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:22:55.049 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:55.312 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:22:55.506 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T00:22:55.523 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:55.667 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:55.899 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:56.246 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:22:56.246 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:56.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:22:56.247 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-20T00:22:56.247 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:56.247 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:22:56.264 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:17 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:22:56.453 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T00:22:57.209 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:17 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:22:57.813 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:57.814 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 163 k 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 587 k 2024-02-20T00:22:57.815 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:22:58.143 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 496 kB/s | 163 kB 00:00 2024-02-20T00:22:58.147 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:58.148 INFO:teuthology.orchestra.run.smithi087.stdout:Total 493 kB/s | 163 kB 00:00 2024-02-20T00:22:58.148 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:22:58.156 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:22:58.156 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:22:58.203 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:22:58.206 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:22:58.751 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 163 k 2024-02-20T00:22:58.752 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 587 k 2024-02-20T00:22:58.753 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:22:58.797 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.6 MB/s | 163 kB 00:00 2024-02-20T00:22:58.798 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:22:58.798 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.5 MB/s | 163 kB 00:00 2024-02-20T00:22:58.799 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:22:58.809 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:22:58.810 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:22:58.856 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:22:58.859 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:22:58.907 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:22:58.975 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:22:59.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:22:59.493 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:22:59.869 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:22:59.869 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:59.870 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:22:59.870 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-20T00:22:59.870 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:22:59.870 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:00.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-02-20T00:23:00.442 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:00.871 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:21 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:00.873 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:01.234 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:01.234 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:01.234 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:01.234 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-20T00:23:01.235 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:01.235 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:01.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-02-20T00:23:02.244 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:22 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:02.389 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:02.436 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:02.437 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:23:02.437 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:02.538 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T00:23:03.287 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:24 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:03.737 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:03.785 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:03.786 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:23:03.786 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:03.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T00:23:04.630 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:24 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:04.829 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:23:04.830 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-20T00:23:04.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:04.832 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.0 M 2024-02-20T00:23:04.832 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 15 M 2024-02-20T00:23:04.832 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:05.337 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 611 kB/s | 24 kB 00:00 2024-02-20T00:23:05.362 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 138 kB/s | 8.9 kB 00:00 2024-02-20T00:23:05.421 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 22 MB/s | 2.7 MB 00:00 2024-02-20T00:23:05.496 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2024-02-20T00:23:05.530 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.8 MB/s | 196 kB 00:00 2024-02-20T00:23:05.705 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 116 kB/s | 43 kB 00:00 2024-02-20T00:23:05.708 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:05.708 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-02-20T00:23:05.719 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:05.729 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:05.729 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:05.888 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:05.891 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:06.161 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:06.176 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-20T00:23:06.177 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-20T00:23:06.178 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:06.179 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.0 M 2024-02-20T00:23:06.179 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 15 M 2024-02-20T00:23:06.179 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:06.376 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T00:23:06.577 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 189 kB/s | 8.9 kB 00:00 2024-02-20T00:23:06.594 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 373 kB/s | 24 kB 00:00 2024-02-20T00:23:06.613 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T00:23:06.652 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 22 MB/s | 2.7 MB 00:00 2024-02-20T00:23:06.805 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T00:23:06.836 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-02-20T00:23:06.894 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 134 kB/s | 43 kB 00:00 2024-02-20T00:23:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-20T00:23:07.045 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 501 kB/s | 196 kB 00:00 2024-02-20T00:23:07.047 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:07.047 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-02-20T00:23:07.057 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:07.067 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:07.067 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:07.235 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:07.238 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:07.508 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:07.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T00:23:07.923 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-20T00:23:07.933 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T00:23:08.109 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T00:23:08.213 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-20T00:23:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T00:23:08.509 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T00:23:08.917 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T00:23:08.917 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:08.918 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:09.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T00:23:09.206 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-20T00:23:09.452 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:09.723 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:09.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:23:09.723 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-20T00:23:09.724 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-20T00:23:09.724 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T00:23:09.724 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T00:23:09.788 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:10.143 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:10.257 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T00:23:11.012 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:31 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:11.327 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:11.328 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-20T00:23:11.328 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:23:11.328 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-20T00:23:11.328 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:11.328 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T00:23:11.329 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:11.330 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:11.330 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-20T00:23:11.330 INFO:teuthology.orchestra.run.smithi087.stdout:Install 8 Packages 2024-02-20T00:23:11.330 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:11.331 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2024-02-20T00:23:11.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 210 M 2024-02-20T00:23:11.331 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:11.920 INFO:teuthology.orchestra.run.smithi087.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 62 MB/s | 8.6 MB 00:00 2024-02-20T00:23:12.170 INFO:teuthology.orchestra.run.smithi087.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2024-02-20T00:23:12.236 INFO:teuthology.orchestra.run.smithi087.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-02-20T00:23:12.262 INFO:teuthology.orchestra.run.smithi087.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-20T00:23:12.337 INFO:teuthology.orchestra.run.smithi087.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.7 MB/s | 4.9 MB 00:00 2024-02-20T00:23:12.549 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:12.550 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:12.551 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================================ 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout:Install 8 Packages 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 37 M 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 210 M 2024-02-20T00:23:12.552 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:12.754 INFO:teuthology.orchestra.run.smithi087.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 27 MB/s | 14 MB 00:00 2024-02-20T00:23:12.788 INFO:teuthology.orchestra.run.smithi087.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 383 kB/s | 172 kB 00:00 2024-02-20T00:23:12.971 INFO:teuthology.orchestra.run.smithi087.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 910 kB/s | 645 kB 00:00 2024-02-20T00:23:12.972 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:12.972 INFO:teuthology.orchestra.run.smithi087.stdout:Total 23 MB/s | 37 MB 00:01 2024-02-20T00:23:13.067 INFO:teuthology.orchestra.run.smithi195.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-02-20T00:23:13.199 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:13.218 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:13.218 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:13.251 INFO:teuthology.orchestra.run.smithi195.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-02-20T00:23:13.324 INFO:teuthology.orchestra.run.smithi195.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.0 MB/s | 225 kB 00:00 2024-02-20T00:23:13.393 INFO:teuthology.orchestra.run.smithi195.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-02-20T00:23:13.541 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:13.545 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:13.585 INFO:teuthology.orchestra.run.smithi195.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 7.1 MB/s | 4.7 MB 00:00 2024-02-20T00:23:13.744 INFO:teuthology.orchestra.run.smithi195.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 34 MB/s | 14 MB 00:00 2024-02-20T00:23:14.019 INFO:teuthology.orchestra.run.smithi195.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2024-02-20T00:23:14.128 INFO:teuthology.orchestra.run.smithi195.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 318 kB/s | 172 kB 00:00 2024-02-20T00:23:14.128 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:14.128 INFO:teuthology.orchestra.run.smithi195.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-20T00:23:14.179 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:14.344 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:23:14.359 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:14.377 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:14.377 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:14.477 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:23:14.640 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:23:14.703 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:14.706 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:15.229 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:23:15.306 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:15.464 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:23:15.621 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:23:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:15.787 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:23:16.323 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:16.382 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:23:16.502 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:23:16.556 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:17.162 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:17.194 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:23:17.555 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:23:17.835 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:23:17.890 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:23:18.584 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:23:18.892 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:23:19.139 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:23:19.582 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:23:20.874 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T00:23:20.901 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:23:21.343 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:23:21.670 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T00:23:21.671 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T00:23:22.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T00:23:22.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:22.077 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:22.077 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-20T00:23:22.077 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:22.078 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:22.193 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-02-20T00:23:22.220 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T00:23:22.392 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:23:22.734 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T00:23:22.735 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T00:23:22.948 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:43 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:23.300 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T00:23:23.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:23.300 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:23.300 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-20T00:23:23.300 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:23.301 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:23.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-02-20T00:23:24.163 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:44 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:24.490 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:24.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:24.491 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:23:24.491 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:24.491 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T00:23:24.492 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-02-20T00:23:24.493 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:24.494 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-02-20T00:23:24.494 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2024-02-20T00:23:24.494 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:25.280 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 418 kB/s | 27 kB 00:00 2024-02-20T00:23:25.337 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-02-20T00:23:25.362 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 136 kB/s | 20 kB 00:00 2024-02-20T00:23:25.387 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-02-20T00:23:25.421 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2024-02-20T00:23:25.705 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2024-02-20T00:23:25.714 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:25.716 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:25.716 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:23:25.716 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:25.716 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:25.716 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T00:23:25.717 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T00:23:25.718 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-02-20T00:23:25.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:25.720 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.9 M 2024-02-20T00:23:25.721 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2024-02-20T00:23:25.721 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:25.855 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 315 kB/s | 155 kB 00:00 2024-02-20T00:23:25.880 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 72 kB/s | 33 kB 00:00 2024-02-20T00:23:26.027 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 601 kB/s | 27 kB 00:00 2024-02-20T00:23:26.077 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 211 kB/s | 20 kB 00:00 2024-02-20T00:23:26.103 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 221 kB/s | 27 kB 00:00 2024-02-20T00:23:26.128 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-02-20T00:23:26.145 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.6 MB/s | 110 kB 00:00 2024-02-20T00:23:26.147 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 37 kB/s | 16 kB 00:00 2024-02-20T00:23:26.364 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-02-20T00:23:26.404 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 128 kB/s | 33 kB 00:00 2024-02-20T00:23:26.412 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 146 kB/s | 81 kB 00:00 2024-02-20T00:23:26.437 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2024-02-20T00:23:26.473 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2024-02-20T00:23:26.504 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 162 kB/s | 16 kB 00:00 2024-02-20T00:23:26.563 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 337 kB/s | 155 kB 00:00 2024-02-20T00:23:26.763 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 249 kB/s | 81 kB 00:00 2024-02-20T00:23:26.872 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2024-02-20T00:23:26.889 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 197 kB/s | 64 kB 00:00 2024-02-20T00:23:26.922 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.8 MB/s | 1.2 MB 00:00 2024-02-20T00:23:26.923 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:26.923 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2024-02-20T00:23:26.981 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:26.997 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:26.997 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:27.086 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:27.089 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:27.423 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 774 kB/s | 1.2 MB 00:01 2024-02-20T00:23:27.426 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:27.426 INFO:teuthology.orchestra.run.smithi087.stdout:Total 667 kB/s | 1.9 MB 00:02 2024-02-20T00:23:27.484 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:27.500 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:27.500 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:27.587 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:27.590 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:27.604 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:27.823 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T00:23:27.996 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T00:23:28.096 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:28.323 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T00:23:28.336 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T00:23:28.525 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T00:23:28.536 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T00:23:28.727 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T00:23:28.777 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T00:23:28.974 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T00:23:28.976 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T00:23:29.156 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T00:23:29.261 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T00:23:29.396 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T00:23:29.527 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T00:23:29.655 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T00:23:29.922 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T00:23:30.555 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T00:23:30.818 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T00:23:30.949 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T00:23:31.044 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T00:23:31.219 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T00:23:31.232 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T00:23:31.378 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:23:31.424 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T00:23:31.616 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T00:23:31.663 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:23:31.663 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-20T00:23:31.663 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T00:23:31.663 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T00:23:31.664 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T00:23:31.758 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:23:32.051 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T00:23:32.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T00:23:32.124 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:32.125 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:32.239 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T00:23:32.516 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T00:23:32.516 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:32.516 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:32.516 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-20T00:23:32.516 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:32.517 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:32.630 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T00:23:32.991 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:52 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:33.380 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:34.546 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-20T00:23:34.547 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-02-20T00:23:34.548 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:34.549 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.9 M 2024-02-20T00:23:34.549 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2024-02-20T00:23:34.549 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:34.935 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:34.936 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:34.936 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-20T00:23:34.936 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:34.936 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-20T00:23:34.937 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:34.938 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.9 M 2024-02-20T00:23:34.938 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2024-02-20T00:23:34.938 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:34.972 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2024-02-20T00:23:35.089 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 3.3 MB/s | 578 kB 00:00 2024-02-20T00:23:35.134 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 131 kB/s | 21 kB 00:00 2024-02-20T00:23:35.238 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.4 MB/s | 89 kB 00:00 2024-02-20T00:23:35.298 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2024-02-20T00:23:35.324 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 258 kB/s | 49 kB 00:00 2024-02-20T00:23:35.347 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 3.9 MB/s | 578 kB 00:00 2024-02-20T00:23:35.357 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2024-02-20T00:23:35.358 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:35.358 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.2 MB/s | 5.9 MB 00:00 2024-02-20T00:23:35.408 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:35.417 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:35.417 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:35.516 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:35.519 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:35.530 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-02-20T00:23:35.555 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 67 kB/s | 21 kB 00:00 2024-02-20T00:23:35.765 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 209 kB/s | 49 kB 00:00 2024-02-20T00:23:35.795 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:35.931 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 608 kB/s | 355 kB 00:00 2024-02-20T00:23:35.932 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:35.932 INFO:teuthology.orchestra.run.smithi087.stdout:Total 5.9 MB/s | 5.9 MB 00:00 2024-02-20T00:23:35.984 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:35.994 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:35.994 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T00:23:36.101 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:36.105 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:36.321 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T00:23:36.407 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:36.707 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T00:23:36.994 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T00:23:37.223 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T00:23:37.589 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T00:23:37.807 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T00:23:37.908 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T00:23:38.027 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:38.245 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T00:23:38.297 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:38.298 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:23:38.298 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T00:23:38.298 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T00:23:38.298 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T00:23:38.298 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T00:23:38.470 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T00:23:38.686 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T00:23:38.686 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:38.687 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:38.707 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:38.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T00:23:39.001 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:39.391 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:39.514 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-02-20T00:23:39.588 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:59 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:40.265 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:01 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:41.143 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:41.144 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:41.145 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-02-20T00:23:41.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:41.145 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 968 k 2024-02-20T00:23:41.145 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 3.2 M 2024-02-20T00:23:41.145 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:41.242 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-20T00:23:41.567 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 218 kB/s | 83 kB 00:00 2024-02-20T00:23:41.567 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:41.568 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-02-20T00:23:41.577 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:41.598 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:41.598 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:41.687 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:41.690 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:41.810 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-02-20T00:23:41.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:41.812 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 968 k 2024-02-20T00:23:41.812 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 3.2 M 2024-02-20T00:23:41.812 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:41.867 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:42.050 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-20T00:23:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T00:23:42.343 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:23:42.383 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 215 kB/s | 83 kB 00:00 2024-02-20T00:23:42.384 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:42.384 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.7 MB/s | 968 kB 00:00 2024-02-20T00:23:42.394 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:42.415 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:42.415 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:42.502 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:42.505 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:42.703 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:42.737 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:23:42.737 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-20T00:23:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:43.129 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:43.182 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:23:43.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-02-20T00:23:43.557 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:23:43.557 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-20T00:23:43.947 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T00:23:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:44.071 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T00:23:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:44.071 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:44.127 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:04 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:44.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-02-20T00:23:44.942 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:05 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:45.692 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:45.693 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:45.693 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:45.693 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 131 k 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 439 k 2024-02-20T00:23:45.694 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:45.749 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-02-20T00:23:45.749 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:45.750 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-02-20T00:23:45.750 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:45.755 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:45.755 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:45.807 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:45.807 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:45.925 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:46.167 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:46.458 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:46.489 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:23:46.490 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:46.491 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 131 k 2024-02-20T00:23:46.491 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 439 k 2024-02-20T00:23:46.491 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:46.545 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-02-20T00:23:46.546 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:46.546 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-02-20T00:23:46.546 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:46.551 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:46.551 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:46.600 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:46.604 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:46.715 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:46.827 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:46.828 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:23:46.828 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:46.828 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:47.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-02-20T00:23:47.821 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:07 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:47.876 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:48.575 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:48.932 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:48.933 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:48.933 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:48.933 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:23:48.933 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:48.933 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:49.201 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-02-20T00:23:49.315 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:49.363 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:49.364 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:23:49.364 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:49.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-02-20T00:23:49.967 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:10 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:50.652 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:10 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:51.550 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:51.596 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:51.598 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:23:51.598 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:51.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-02-20T00:23:52.210 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:52.210 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:52.210 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:23:52.211 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:52.212 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 24 k 2024-02-20T00:23:52.212 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 94 k 2024-02-20T00:23:52.212 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:23:52.255 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 567 kB/s | 24 kB 00:00 2024-02-20T00:23:52.255 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:52.255 INFO:teuthology.orchestra.run.smithi195.stdout:Total 551 kB/s | 24 kB 00:00 2024-02-20T00:23:52.256 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:23:52.259 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:23:52.259 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:23:52.261 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:23:52.264 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:23:52.318 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:23:52.460 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:13 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:52.513 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:52.766 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:23:53.121 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:53.223 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-02-20T00:23:53.990 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:13 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:54.022 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 24 k 2024-02-20T00:23:54.023 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 94 k 2024-02-20T00:23:54.024 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:23:54.212 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 129 kB/s | 24 kB 00:00 2024-02-20T00:23:54.258 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Total 128 kB/s | 24 kB 00:00 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:23:54.259 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:23:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:23:55.010 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:55.270 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:55.498 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:55.545 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:55.546 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:23:55.546 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:55.628 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:23:55.629 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:55.629 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:23:55.629 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:23:55.629 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:23:55.629 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:55.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-02-20T00:23:56.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-02-20T00:23:56.495 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:17 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:56.768 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:16 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:23:57.992 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:58.038 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:23:58.039 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:23:58.039 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:23:58.145 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-02-20T00:23:58.284 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:23:58.331 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:23:58.332 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:23:58.332 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:23:58.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-02-20T00:23:58.899 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:19 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:23:59.215 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:19 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:00.403 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:00.449 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:00.451 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:24:00.451 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:00.550 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-02-20T00:24:00.717 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:00.763 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:00.765 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:24:00.765 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:00.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-02-20T00:24:01.300 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:22 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:01.616 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:21 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:02.803 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:02.849 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:02.850 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:24:02.850 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:02.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-02-20T00:24:03.087 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:03.133 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:03.135 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:24:03.135 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:03.236 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-02-20T00:24:03.697 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:24 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:03.989 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:23 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:05.212 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:05.258 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:24:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:05.470 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-02-20T00:24:05.478 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:05.525 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:05.526 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:24:05.526 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:05.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-02-20T00:24:06.216 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:27 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:06.383 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:26 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:07.726 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:07.771 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:07.773 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:24:07.773 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:07.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-02-20T00:24:07.915 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:07.961 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:07.962 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-02-20T00:24:07.962 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:08.068 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-02-20T00:24:08.629 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:29 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:08.822 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:28 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:10.129 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:24:10.176 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:10.177 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-20T00:24:10.177 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:10.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-02-20T00:24:10.374 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:24:10.375 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:10.376 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 89 k 2024-02-20T00:24:10.376 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 298 k 2024-02-20T00:24:10.376 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:24:10.420 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-02-20T00:24:10.420 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:10.421 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-02-20T00:24:10.421 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:24:10.431 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:24:10.431 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:24:10.480 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:24:10.483 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:24:10.592 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:24:10.788 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:11.046 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:32 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:11.072 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:11.448 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:11.448 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:11.449 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:24:11.449 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:11.449 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:11.449 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:11.676 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-02-20T00:24:12.447 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:32 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:24:12.618 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:12.619 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:24:12.619 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:12.619 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 89 k 2024-02-20T00:24:12.619 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 298 k 2024-02-20T00:24:12.619 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:24:12.666 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-20T00:24:12.667 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:12.667 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-20T00:24:12.667 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:24:12.677 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:24:12.677 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:24:12.725 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:24:12.728 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:24:12.830 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:24:13.018 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:13.297 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:13.653 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:13.856 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-02-20T00:24:13.990 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:13.990 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:24:13.991 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:13.992 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.7 M 2024-02-20T00:24:13.992 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 18 M 2024-02-20T00:24:13.992 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:24:14.082 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2024-02-20T00:24:14.083 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:14.083 INFO:teuthology.orchestra.run.smithi195.stdout:Total 40 MB/s | 3.7 MB 00:00 2024-02-20T00:24:14.084 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:24:14.095 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:24:14.095 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:24:14.158 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:24:14.161 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:24:14.621 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:35 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:14.627 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:24:14.822 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:15.223 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:15.619 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:15.834 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-02-20T00:24:16.161 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:16.162 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:24:16.163 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:16.163 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.7 M 2024-02-20T00:24:16.163 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 18 M 2024-02-20T00:24:16.163 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:24:16.251 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-02-20T00:24:16.252 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:16.252 INFO:teuthology.orchestra.run.smithi087.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-02-20T00:24:16.253 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:24:16.264 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:24:16.265 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:24:16.327 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:24:16.330 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:24:16.599 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:36 ago on Tue 20 Feb 2024 12:21:40 AM UTC. 2024-02-20T00:24:16.790 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:24:16.986 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:17.399 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:17.785 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:17.785 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:17.786 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:24:17.786 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:17.786 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:17.786 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:17.999 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-02-20T00:24:18.149 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-02-20T00:24:18.149 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:18.149 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-02-20T00:24:18.150 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:18.151 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 173 k 2024-02-20T00:24:18.151 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 649 k 2024-02-20T00:24:18.151 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-02-20T00:24:18.188 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.6 MB/s | 173 kB 00:00 2024-02-20T00:24:18.189 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:18.189 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.4 MB/s | 173 kB 00:00 2024-02-20T00:24:18.189 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-02-20T00:24:18.200 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-02-20T00:24:18.200 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-02-20T00:24:18.250 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-02-20T00:24:18.252 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-02-20T00:24:18.377 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-02-20T00:24:18.597 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:18.762 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:39 ago on Tue 20 Feb 2024 12:21:39 AM UTC. 2024-02-20T00:24:18.892 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:24:19.299 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-02-20T00:24:19.490 DEBUG:teuthology.parallel:result is None 2024-02-20T00:24:20.315 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-20T00:24:20.315 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:20.315 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-20T00:24:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:20.317 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 173 k 2024-02-20T00:24:20.317 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 649 k 2024-02-20T00:24:20.317 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-20T00:24:20.370 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 173 kB 00:00 2024-02-20T00:24:20.371 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:24:20.371 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.1 MB/s | 173 kB 00:00 2024-02-20T00:24:20.371 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-20T00:24:20.382 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-20T00:24:20.382 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-20T00:24:20.431 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-20T00:24:20.434 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-20T00:24:20.548 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-20T00:24:20.770 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:21.053 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:21.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:24:21.684 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:21.684 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-20T00:24:21.684 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:24:21.685 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-20T00:24:21.685 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-20T00:24:21.925 DEBUG:teuthology.parallel:result is None 2024-02-20T00:24:21.925 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:24:21.925 INFO:teuthology.packaging:ref: None 2024-02-20T00:24:21.926 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:24:21.926 INFO:teuthology.packaging:branch: None 2024-02-20T00:24:21.926 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:24:21.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:24:22.064 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T00:24:22.086 INFO:teuthology.orchestra.run.smithi087.stdout:16.2.4-0.el8 2024-02-20T00:24:22.087 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-20T00:24:22.087 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-20T00:24:22.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:24:22.089 INFO:teuthology.packaging:ref: None 2024-02-20T00:24:22.089 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:24:22.089 INFO:teuthology.packaging:branch: None 2024-02-20T00:24:22.089 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:24:22.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:24:22.224 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T00:24:22.251 INFO:teuthology.orchestra.run.smithi195.stdout:16.2.4-0.el8 2024-02-20T00:24:22.252 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-20T00:24:22.252 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-20T00:24:22.253 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T00:24:22.254 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:24:22.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T00:24:22.281 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:24:22.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T00:24:22.323 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T00:24:22.324 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:24:22.324 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T00:24:22.351 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T00:24:22.417 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:24:22.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T00:24:22.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T00:24:22.521 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T00:24:22.521 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:24:22.521 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T00:24:22.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T00:24:22.614 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:24:22.614 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T00:24:22.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T00:24:22.712 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T00:24:22.713 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:24:22.713 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T00:24:22.739 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T00:24:22.806 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:24:22.806 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T00:24:22.835 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T00:24:22.904 INFO:teuthology.run_tasks:Running task print... 2024-02-20T00:24:22.913 INFO:teuthology.task.print:**** done install task... 2024-02-20T00:24:22.913 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T00:24:22.984 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'} 2024-02-20T00:24:22.985 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-20T00:24:22.985 INFO:tasks.cephadm:Cluster fsid is 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:22.985 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T00:24:22.985 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T00:24:22.985 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi087': '172.21.15.87', 'mon.smithi195': '172.21.15.195'} 2024-02-20T00:24:22.985 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T00:24:22.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname $(hostname -s) 2024-02-20T00:24:23.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-02-20T00:24:23.041 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-20T00:24:23.042 DEBUG:teuthology.orchestra.run.smithi087:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.257 INFO:teuthology.orchestra.run.smithi087.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 20 00:24 /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.257 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.440 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 20 00:24 /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.440 DEBUG:teuthology.orchestra.run.smithi087:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.459 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T00:24:23.489 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2024-02-20T00:24:23.526 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-02-20T00:24:23.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /etc/ceph 2024-02-20T00:24:23.591 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-02-20T00:24:23.623 INFO:tasks.cephadm:Writing seed config... 2024-02-20T00:24:23.624 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T00:24:23.624 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T00:24:23.624 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T00:24:23.625 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T00:24:23.626 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T00:24:23.627 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T00:24:23.627 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T00:24:23.627 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T00:24:23.627 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T00:24:23.627 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T00:24:23.627 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:24:23.627 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T00:24:23.649 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6575da04-cf86-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T00:24:23.649 DEBUG:teuthology.orchestra.run.smithi087:mon.smithi087> sudo journalctl -f -n 0 -u ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service 2024-02-20T00:24:23.692 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T00:24:23.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 6575da04-cf86-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.87 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:24:23.718 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:-- Logs begin at Tue 2024-02-20 00:14:49 UTC. -- 2024-02-20T00:24:23.960 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:24:24.904 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:24:24.919 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying podman|docker is present... 2024-02-20T00:24:24.920 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:24:24.939 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:24:24.947 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying lvm2 is present... 2024-02-20T00:24:24.948 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying time synchronization is in place... 2024-02-20T00:24:24.948 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled chrony.service 2024-02-20T00:24:24.955 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T00:24:24.955 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active chrony.service 2024-02-20T00:24:24.962 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:24:24.962 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-20T00:24:24.968 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: enabled 2024-02-20T00:24:24.969 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active chronyd.service 2024-02-20T00:24:24.974 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: active 2024-02-20T00:24:24.975 INFO:teuthology.orchestra.run.smithi087.stderr:Unit chronyd.service is enabled and running 2024-02-20T00:24:24.975 INFO:teuthology.orchestra.run.smithi087.stderr:Repeating the final host check... 2024-02-20T00:24:24.975 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:24:24.998 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:24:25.006 INFO:teuthology.orchestra.run.smithi087.stderr:podman|docker (/bin/podman) is present 2024-02-20T00:24:25.006 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl is present 2024-02-20T00:24:25.007 INFO:teuthology.orchestra.run.smithi087.stderr:lvcreate is present 2024-02-20T00:24:25.007 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled chrony.service 2024-02-20T00:24:25.012 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T00:24:25.013 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active chrony.service 2024-02-20T00:24:25.018 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:24:25.019 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-20T00:24:25.024 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: enabled 2024-02-20T00:24:25.024 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active chronyd.service 2024-02-20T00:24:25.029 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: active 2024-02-20T00:24:25.031 INFO:teuthology.orchestra.run.smithi087.stderr:Unit chronyd.service is enabled and running 2024-02-20T00:24:25.031 INFO:teuthology.orchestra.run.smithi087.stderr:Host looks OK 2024-02-20T00:24:25.031 INFO:teuthology.orchestra.run.smithi087.stderr:Cluster fsid: 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:25.031 INFO:teuthology.orchestra.run.smithi087.stderr:Acquiring lock 139849865869800 on /run/cephadm/6575da04-cf86-11ee-95bb-87774f69a715.lock 2024-02-20T00:24:25.032 INFO:teuthology.orchestra.run.smithi087.stderr:Lock 139849865869800 acquired on /run/cephadm/6575da04-cf86-11ee-95bb-87774f69a715.lock 2024-02-20T00:24:25.032 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying IP 172.21.15.87 port 3300 ... 2024-02-20T00:24:25.032 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying IP 172.21.15.87 port 6789 ... 2024-02-20T00:24:25.032 INFO:teuthology.orchestra.run.smithi087.stderr:Base mon IP is 172.21.15.87, final addrv is [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-02-20T00:24:25.032 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /sbin/ip route ls 2024-02-20T00:24:25.036 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.87 metric 100 2024-02-20T00:24:25.036 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.87 metric 100 2024-02-20T00:24:25.037 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /sbin/ip -6 route ls 2024-02-20T00:24:25.037 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T00:24:25.037 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-20T00:24:25.037 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-02-20T00:24:25.038 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /sbin/ip -6 addr ls 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d5f5/64 scope link 2024-02-20T00:24:25.040 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T00:24:25.042 INFO:teuthology.orchestra.run.smithi087.stderr:Mon IP 172.21.15.87 is in CIDR network 172.21.0.0/20 2024-02-20T00:24:25.042 INFO:teuthology.orchestra.run.smithi087.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T00:24:25.042 INFO:teuthology.orchestra.run.smithi087.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-20T00:24:25.042 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-20T00:24:25.130 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-20T00:24:31.704 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Getting image source signatures 2024-02-20T00:24:31.716 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-20T00:24:31.716 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-20T00:24:43.574 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-20T00:24:43.758 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Writing manifest to image destination 2024-02-20T00:24:44.034 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-20T00:24:44.045 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-20T00:24:44.657 INFO:teuthology.orchestra.run.smithi087.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-20T00:24:45.261 INFO:teuthology.orchestra.run.smithi087.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-20T00:24:45.261 INFO:teuthology.orchestra.run.smithi087.stderr:Extracting ceph user uid/gid from container image... 2024-02-20T00:24:45.261 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-20T00:24:45.770 INFO:teuthology.orchestra.run.smithi087.stderr:stat: 167 167 2024-02-20T00:24:46.271 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial keys... 2024-02-20T00:24:46.271 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:24:46.785 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQBO8dNl8IAzKRAA+7VF2PI5/fPF4UNuNGiviw== 2024-02-20T00:24:47.369 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:24:47.932 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQBP8dNlsP4mMRAArAi+68nVkZt4Nl/h+ECWDA== 2024-02-20T00:24:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:24:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQBQ8dNlxtSANBAA2wUAz/fcXcl+HqUoIorgBw== 2024-02-20T00:24:49.523 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial monmap... 2024-02-20T00:24:49.523 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmps42_1hx8:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 6575da04-cf86-11ee-95bb-87774f69a715 --addv smithi087 [v2:172.21.15.87:3300,v1:172.21.15.87:6789] /tmp/monmap 2024-02-20T00:24:50.019 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T00:24:50.019 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:50.020 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T00:24:50.548 INFO:teuthology.orchestra.run.smithi087.stderr:monmaptool for smithi087 [v2:172.21.15.87:3300,v1:172.21.15.87:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T00:24:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: set fsid to 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T00:24:50.549 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:24:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:Creating mon... 2024-02-20T00:24:50.549 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/mon.smithi087:/var/lib/ceph/mon/ceph-smithi087:z -v /tmp/ceph-tmpbvjuv8ai:/tmp/keyring:z -v /tmp/ceph-tmps42_1hx8:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi087 --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-20T00:24:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.023+0000 7fdd1057e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T00:24:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.023+0000 7fdd1057e700 1 imported monmap: 2024-02-20T00:24:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-20T00:24:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: fsid 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: last_changed 2024-02-20T00:24:49.938605+0000 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: created 2024-02-20T00:24:49.938605+0000 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.smithi087 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.023+0000 7fdd1057e700 0 /usr/bin/ceph-mon: set fsid to 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T00:24:51.129 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Compile date May 13 2021 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: DB SUMMARY 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi087/store.db dir, Total Num: 0, files: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi087/store.db: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.create_if_missing: 1 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.env: 0x561b1888a1c0 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.fs: Posix File System 2024-02-20T00:24:51.130 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.info_log: 0x561b19710ec0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.statistics: (nil) 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.use_fsync: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T00:24:51.131 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.db_log_dir: 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi087/store.db 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.write_buffer_manager: 0x561b19717530 2024-02-20T00:24:51.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.unordered_write: 0 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T00:24:51.133 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.row_cache: None 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.wal_filter: None 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T00:24:51.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_open_files: -1 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T00:24:51.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Compression algorithms supported: 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kZSTD supported: 0 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi087/store.db/MANIFEST-000001 2024-02-20T00:24:51.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.merge_operator: 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compaction_filter: None 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561b19629cb8) 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T00:24:51.137 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache: 0x561b1965ed10 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-20T00:24:51.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-20T00:24:51.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compression: NoCompression 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.num_levels: 7 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T00:24:51.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.024+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T00:24:51.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T00:24:51.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T00:24:51.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.ttl: 2592000 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi087/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.025+0000 7fdd1057e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.026+0000 7fdd1057e700 4 rocksdb: DB pointer 0x561b19723800 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.026+0000 7fdcf9417700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.026+0000 7fdcf9417700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.145 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:24:51.146 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:51.147 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:24:51.148 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:24:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-20T00:24:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.027+0000 7fdd1057e700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-20T00:24:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.027+0000 7fdd1057e700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-20T00:24:51.149 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:24:51.027+0000 7fdd1057e700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi087 for mon.smithi087 2024-02-20T00:24:51.642 INFO:teuthology.orchestra.run.smithi087.stderr:create mon.smithi087 on 2024-02-20T00:24:51.643 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:51.649 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl disable ceph.target 2024-02-20T00:24:51.656 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-20T00:24:51.743 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl enable ceph.target 2024-02-20T00:24:51.751 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T00:24:51.832 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl start ceph.target 2024-02-20T00:24:51.839 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl enable ceph-6575da04-cf86-11ee-95bb-87774f69a715.target 2024-02-20T00:24:51.845 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target. 2024-02-20T00:24:51.845 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target. 2024-02-20T00:24:51.923 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715.target 2024-02-20T00:24:51.930 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl daemon-reload 2024-02-20T00:24:52.015 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:24:52.038 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl reset-failed ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:24:52.043 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to reset failed state of unit ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service: Unit ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service not loaded. 2024-02-20T00:24:52.044 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl enable ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:24:52.050 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target.wants/ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service → /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715@.service. 2024-02-20T00:24:52.136 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:24:52.762 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:24:52.770 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-20T00:24:52.770 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:24:52.777 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:24:52.778 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-20T00:24:52.778 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T00:24:52.778 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-20T00:24:52.779 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mon to start... 2024-02-20T00:24:52.779 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mon... 2024-02-20T00:24:52.779 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/mon.smithi087:/var/lib/ceph/mon/ceph-smithi087:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-20T00:24:53.019 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:52 smithi087 bash[37648]: 1ad4248765874e3f53d94e2c17ab47b8565ed54777f22036146d40e5c4e9dc0a 2024-02-20T00:24:53.019 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:52 smithi087 systemd[1]: Started Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715. 2024-02-20T00:24:53.337 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bdd5d6700 1 Processor -- start 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bdd5d6700 1 -- start start 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bdd5d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bdd5d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd80fc7b0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bd6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bd6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53562/0 (socket says 172.21.15.87:53562) 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.331+0000 7f2bd6ffd700 1 -- 172.21.15.87:0/381909960 learned_addr learned my addr 172.21.15.87:0/381909960 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.332+0000 7f2bd6ffd700 1 -- 172.21.15.87:0/381909960 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd80fc8f0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.332+0000 7f2bd6ffd700 1 --2- 172.21.15.87:0/381909960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2bc800b5f0 tx=0x7f2bc8006ee0).ready entity=mon.0 client_cookie=60366589a5b0612e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.332+0000 7f2bd5ffb700 1 -- 172.21.15.87:0/381909960 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc800d600 con 0x7f2bd80fbea0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.332+0000 7f2bd5ffb700 1 -- 172.21.15.87:0/381909960 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc800d760 con 0x7f2bd80fbea0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.332+0000 7f2bd5ffb700 1 -- 172.21.15.87:0/381909960 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc800da30 con 0x7f2bd80fbea0 2024-02-20T00:24:53.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/381909960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 msgr2=0x7f2bd80fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 --2- 172.21.15.87:0/381909960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2bc800b5f0 tx=0x7f2bc8006ee0).stop 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/381909960 shutdown_connections 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 --2- 172.21.15.87:0/381909960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd80fc2c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/381909960 >> 172.21.15.87:0/381909960 conn(0x7f2bd80f76a0 msgr2=0x7f2bd80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/381909960 shutdown_connections 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/381909960 wait complete. 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 Processor -- start 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.333+0000 7f2bdd5d6700 1 -- start start 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bdd5d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bdd5d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd810a810 con 0x7f2bd80fbea0 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bd6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bd6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53574/0 (socket says 172.21.15.87:53574) 2024-02-20T00:24:53.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bd6ffd700 1 -- 172.21.15.87:0/3906786075 learned_addr learned my addr 172.21.15.87:0/3906786075 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bd6ffd700 1 -- 172.21.15.87:0/3906786075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc800b200 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bd6ffd700 1 --2- 172.21.15.87:0/3906786075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2bc8006c70 tx=0x7f2bc8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc8011b40 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc8011ca0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd810aa10 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc801d430 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd810aea0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.334+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2bc800f070 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.335+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2bc8027460 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.335+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd8045ad0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.340 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.337+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2bc8016070 con 0x7f2bd80fbea0 2024-02-20T00:24:53.405 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.403+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2bd8103eb0 con 0x7f2bd80fbea0 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.403+0000 7f2bbf7fe700 1 -- 172.21.15.87:0/3906786075 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f2bc801d640 con 0x7f2bd80fbea0 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: cluster: 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: id: 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: services: 2024-02-20T00:24:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi087 (age 0.698218s) 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: data: 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: pgs: 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 msgr2=0x7f2bd810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 --2- 172.21.15.87:0/3906786075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2bc8006c70 tx=0x7f2bc8016000).stop 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 shutdown_connections 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 --2- 172.21.15.87:0/3906786075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bd80fbea0 0x7f2bd810a320 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:53.408 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 >> 172.21.15.87:0/3906786075 conn(0x7f2bd80f76a0 msgr2=0x7f2bd80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 shutdown_connections 2024-02-20T00:24:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:53.404+0000 7f2bdd5d6700 1 -- 172.21.15.87:0/3906786075 wait complete. 2024-02-20T00:24:53.992 INFO:teuthology.orchestra.run.smithi087.stderr:mon is available 2024-02-20T00:24:53.992 INFO:teuthology.orchestra.run.smithi087.stderr:Assimilating anything we can from ceph.conf... 2024-02-20T00:24:53.992 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/mon.smithi087:/var/lib/ceph/mon/ceph-smithi087:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi087/config 2024-02-20T00:24:53.994 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: cluster 2024-02-20T00:24:52.706088+0000 mon.smithi087 (mon.0) 2 : cluster [INF] mon.smithi087 is new leader, mons smithi087 in quorum (ranks 0) 2024-02-20T00:24:53.994 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: cluster 2024-02-20T00:24:52.706289+0000 mon.smithi087 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi087=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} 2024-02-20T00:24:53.994 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: cluster 2024-02-20T00:24:52.709277+0000 mon.smithi087 (mon.0) 4 : cluster [DBG] fsmap 2024-02-20T00:24:53.994 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: cluster 2024-02-20T00:24:52.712337+0000 mon.smithi087 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T00:24:53.995 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: cluster 2024-02-20T00:24:52.712704+0000 mon.smithi087 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T00:24:53.995 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: audit 2024-02-20T00:24:53.404278+0000 mon.smithi087 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.87:0/3906786075' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T00:24:54.564 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8432f07700 1 Processor -- start 2024-02-20T00:24:54.564 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8432f07700 1 -- start start 2024-02-20T00:24:54.564 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8432f07700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8432f07700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0fb700 con 0x7f842c0fadf0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8430ca3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8430ca3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53578/0 (socket says 172.21.15.87:53578) 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8430ca3700 1 -- 172.21.15.87:0/3468556916 learned_addr learned my addr 172.21.15.87:0/3468556916 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8430ca3700 1 -- 172.21.15.87:0/3468556916 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0fb840 con 0x7f842c0fadf0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.560+0000 7f8430ca3700 1 --2- 172.21.15.87:0/3468556916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f842000b870 tx=0x7f8420006f70).ready entity=mon.0 client_cookie=34465c4a1d91444f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f842b7fe700 1 -- 172.21.15.87:0/3468556916 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f842000da20 con 0x7f842c0fadf0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f842b7fe700 1 -- 172.21.15.87:0/3468556916 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842000db80 con 0x7f842c0fadf0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f842b7fe700 1 -- 172.21.15.87:0/3468556916 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f842000de50 con 0x7f842c0fadf0 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 -- 172.21.15.87:0/3468556916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 msgr2=0x7f842c0fb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 --2- 172.21.15.87:0/3468556916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f842000b870 tx=0x7f8420006f70).stop 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 -- 172.21.15.87:0/3468556916 shutdown_connections 2024-02-20T00:24:54.565 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 --2- 172.21.15.87:0/3468556916 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c0fadf0 0x7f842c0fb210 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 -- 172.21.15.87:0/3468556916 >> 172.21.15.87:0/3468556916 conn(0x7f842c0f65c0 msgr2=0x7f842c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 -- 172.21.15.87:0/3468556916 shutdown_connections 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.561+0000 7f8432f07700 1 -- 172.21.15.87:0/3468556916 wait complete. 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8432f07700 1 Processor -- start 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8432f07700 1 -- start start 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8432f07700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8432f07700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c10cf30 con 0x7f842c10c620 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8430ca3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8430ca3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53582/0 (socket says 172.21.15.87:53582) 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8430ca3700 1 -- 172.21.15.87:0/2144749767 learned_addr learned my addr 172.21.15.87:0/2144749767 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8430ca3700 1 -- 172.21.15.87:0/2144749767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842000b480 con 0x7f842c10c620 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.562+0000 7f8430ca3700 1 --2- 172.21.15.87:0/2144749767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8420013000 tx=0x7f8420006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8420019450 con 0x7f842c10c620 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84200195b0 con 0x7f842c10c620 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c10d130 con 0x7f842c10c620 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8420019880 con 0x7f842c10c620 2024-02-20T00:24:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c10dc10 con 0x7f842c10c620 2024-02-20T00:24:54.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f842002a070 con 0x7f842c10c620 2024-02-20T00:24:54.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.563+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8420023910 con 0x7f842c10c620 2024-02-20T00:24:54.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.564+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c059100 con 0x7f842c10c620 2024-02-20T00:24:54.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.566+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f84200429c0 con 0x7f842c10c620 2024-02-20T00:24:54.632 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.630+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f842c045ad0 con 0x7f842c10c620 2024-02-20T00:24:54.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.633+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f842000f860 con 0x7f842c10c620 2024-02-20T00:24:54.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.633+0000 7f84297fa700 1 -- 172.21.15.87:0/2144749767 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f8420031020 con 0x7f842c10c620 2024-02-20T00:24:54.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [global] 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: fsid = 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [mgr] 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [osd] 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 msgr2=0x7f842c10ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 --2- 172.21.15.87:0/2144749767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8420013000 tx=0x7f8420006f70).stop 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 shutdown_connections 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 --2- 172.21.15.87:0/2144749767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f842c10c620 0x7f842c10ca40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 >> 172.21.15.87:0/2144749767 conn(0x7f842c0f65c0 msgr2=0x7f842c0f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 shutdown_connections 2024-02-20T00:24:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:54.634+0000 7f8432f07700 1 -- 172.21.15.87:0/2144749767 wait complete. 2024-02-20T00:24:54.996 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: audit 2024-02-20T00:24:54.631214+0000 mon.smithi087 (mon.0) 8 : audit [INF] from='client.? 172.21.15.87:0/2144749767' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T00:24:54.996 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: audit 2024-02-20T00:24:54.634181+0000 mon.smithi087 (mon.0) 9 : audit [INF] from='client.? 172.21.15.87:0/2144749767' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T00:24:55.282 INFO:teuthology.orchestra.run.smithi087.stderr:Generating new minimal ceph.conf... 2024-02-20T00:24:55.282 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/mon.smithi087:/var/lib/ceph/mon/ceph-smithi087:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi087/config 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f402225c700 1 Processor -- start 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f402225c700 1 -- start start 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f402225c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f402225c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0fb590 con 0x7f401c0fac80 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53592/0 (socket says 172.21.15.87:53592) 2024-02-20T00:24:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.739+0000 7f401b7fe700 1 -- 172.21.15.87:0/280591494 learned_addr learned my addr 172.21.15.87:0/280591494 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f401b7fe700 1 -- 172.21.15.87:0/280591494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0fb6d0 con 0x7f401c0fac80 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f401b7fe700 1 --2- 172.21.15.87:0/280591494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f400400b5f0 tx=0x7f4004006ee0).ready entity=mon.0 client_cookie=a6ac204ec24c3a58 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f401a7fc700 1 -- 172.21.15.87:0/280591494 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400400d600 con 0x7f401c0fac80 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f401a7fc700 1 -- 172.21.15.87:0/280591494 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f400400dbf0 con 0x7f401c0fac80 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f401a7fc700 1 -- 172.21.15.87:0/280591494 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4004011cb0 con 0x7f401c0fac80 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f402225c700 1 -- 172.21.15.87:0/280591494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 msgr2=0x7f401c0fb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f402225c700 1 --2- 172.21.15.87:0/280591494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f400400b5f0 tx=0x7f4004006ee0).stop 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f402225c700 1 -- 172.21.15.87:0/280591494 shutdown_connections 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f402225c700 1 --2- 172.21.15.87:0/280591494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c0fb0a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.740+0000 7f402225c700 1 -- 172.21.15.87:0/280591494 >> 172.21.15.87:0/280591494 conn(0x7f401c0f64c0 msgr2=0x7f401c0f88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 -- 172.21.15.87:0/280591494 shutdown_connections 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 -- 172.21.15.87:0/280591494 wait complete. 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 Processor -- start 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 -- start start 2024-02-20T00:24:55.753 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f402225c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c06fc50 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f401b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53594/0 (socket says 172.21.15.87:53594) 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.741+0000 7f401b7fe700 1 -- 172.21.15.87:0/1376519701 learned_addr learned my addr 172.21.15.87:0/1376519701 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f401b7fe700 1 -- 172.21.15.87:0/1376519701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400400b200 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f401b7fe700 1 --2- 172.21.15.87:0/1376519701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4004011ea0 tx=0x7f4004006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400401b570 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f400401bb60 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c06fe50 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f400401a8d0 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c10de30 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f400400f070 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.742+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4004025dd0 con 0x7f401c0fac80 2024-02-20T00:24:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.743+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c045ad0 con 0x7f401c0fac80 2024-02-20T00:24:55.755 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.745+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4004016070 con 0x7f401c0fac80 2024-02-20T00:24:55.812 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.809+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f401c0586e0 con 0x7f401c0fac80 2024-02-20T00:24:55.812 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.809+0000 7f4013fff700 1 -- 172.21.15.87:0/1376519701 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f4004025460 con 0x7f401c0fac80 2024-02-20T00:24:55.813 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 msgr2=0x7f401c10d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:55.813 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 --2- 172.21.15.87:0/1376519701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4004011ea0 tx=0x7f4004006ee0).stop 2024-02-20T00:24:55.814 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 shutdown_connections 2024-02-20T00:24:55.814 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 --2- 172.21.15.87:0/1376519701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f401c0fac80 0x7f401c10d780 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:55.814 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 >> 172.21.15.87:0/1376519701 conn(0x7f401c0f64c0 msgr2=0x7f401c0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:55.814 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 shutdown_connections 2024-02-20T00:24:55.814 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:55.810+0000 7f402225c700 1 -- 172.21.15.87:0/1376519701 wait complete. 2024-02-20T00:24:56.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:55 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: audit 2024-02-20T00:24:55.811495+0000 mon.smithi087 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.87:0/1376519701' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:24:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:Restarting the monitor... 2024-02-20T00:24:56.882 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl restart ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:24:57.281 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:56 smithi087 systemd[1]: Stopping Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715... 2024-02-20T00:24:57.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: debug 2024-02-20T00:24:57.019+0000 7fc77810f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi087 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T00:24:57.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[37663]: debug 2024-02-20T00:24:57.019+0000 7fc77810f700 -1 mon.smithi087@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T00:24:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 podman[37987]: ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087 2024-02-20T00:24:57.908 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 systemd[1]: ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service: Succeeded. 2024-02-20T00:24:57.908 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 systemd[1]: Stopped Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715. 2024-02-20T00:24:57.908 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:57 smithi087 systemd[1]: Starting Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715... 2024-02-20T00:24:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-20T00:24:58.080 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.025+0000 7ff4d6f47700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.025+0000 7ff4d6f47700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.025+0000 7ff4d6f47700 0 pidfile_write: ignore empty --pid-file 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 0 load: jerasure load: lrc load: isa 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Compile date May 13 2021 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: DB SUMMARY 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: CURRENT file: CURRENT 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi087/store.db dir, Total Num: 1, files: 000004.sst 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi087/store.db: 000006.log size: 98708 ; 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.create_if_missing: 0 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T00:24:58.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.env: 0x55e085ed61c0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.fs: Posix File System 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.info_log: 0x55e0873b7960 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.statistics: (nil) 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.use_fsync: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T00:24:58.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.db_log_dir: 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi087/store.db 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T00:24:58.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.write_buffer_manager: 0x55e08802c030 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.unordered_write: 0 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T00:24:58.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.row_cache: None 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.wal_filter: None 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T00:24:58.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_open_files: -1 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T00:24:58.289 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Compression algorithms supported: 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kZSTD supported: 0 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T00:24:58.290 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi087/store.db/MANIFEST-000005 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.merge_operator: 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compaction_filter: None 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e0872cfcb0) 2024-02-20T00:24:58.292 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cache_index_and_filter_blocks: 1 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: pin_top_level_index_and_filter: 1 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: index_type: 0 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: data_block_index_type: 0 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: index_shortening: 1 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: data_block_hash_table_util_ratio: 0.750000 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: hash_index_allow_collision: 1 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: checksum: 1 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: no_block_cache: 0 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_cache: 0x55e087304f10 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_cache_name: BinnedLRUCache 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_cache_options: 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: capacity : 536870912 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: num_shard_bits : 4 2024-02-20T00:24:58.293 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: strict_capacity_limit : 0 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: high_pri_pool_ratio: 0.000 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_cache_compressed: (nil) 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: persistent_cache: (nil) 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_size: 4096 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_size_deviation: 10 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_restart_interval: 16 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: index_block_restart_interval: 1 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: metadata_block_size: 4096 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: partition_filters: 0 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: use_delta_encoding: 1 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: whole_key_filtering: 1 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: verify_compression: 0 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: read_amp_bytes_per_bit: 0 2024-02-20T00:24:58.294 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: format_version: 2 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: enable_index_compression: 1 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: block_align: 0 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression: NoCompression 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.num_levels: 7 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T00:24:58.295 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.029+0000 7ff4d6f47700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T00:24:58.296 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T00:24:58.297 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T00:24:58.298 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.ttl: 2592000 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi087/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708388698032415, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-20T00:24:58.299 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.030+0000 7ff4d6f47700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.031+0000 7ff4d6f47700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.031+0000 7ff4d6f47700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708388698033237, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94282, "table_properties": {"data_size": 92370, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85401, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708388698, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.031+0000 7ff4d6f47700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.031+0000 7ff4d6f47700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708388698033626, "job": 1, "event": "recovery_finished"} 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.032+0000 7ff4d6f47700 4 rocksdb: DB pointer 0x55e0873c9800 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.032+0000 7ff4bdd5f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.032+0000 7ff4bdd5f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** DB Stats ** 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T00:24:58.300 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** Compaction Stats [default] ** 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** Compaction Stats [default] ** 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.301 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Cumulative compaction: 0.00 GB write, 33.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Interval compaction: 0.00 GB write, 33.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** Compaction Stats [default] ** 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:58.302 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** Compaction Stats [default] ** 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.3 0.00 0.00 1 0.001 0 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:24:58.303 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Cumulative compaction: 0.00 GB write, 32.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.033+0000 7ff4d6f47700 0 starting mon.smithi087 rank 0 at public addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] at bind addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi087 fsid 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 1 mon.smithi087@-1(???) e1 preinit fsid 6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).mds e1 new map 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).mds e1 print_map 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: e1 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: legacy client fscid: -1 2024-02-20T00:24:58.304 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: No filesystems configured 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 0 mon.smithi087@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:24:55.811495+0000 mon.smithi087 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.87:0/1376519701' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.034+0000 7ff4d6f47700 1 mon.smithi087@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expand map: {default=false} 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta from 'false' to 'false' 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expanded map: {default=false} 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expand map: {default=info} 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta from 'info' to 'info' 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expanded map: {default=info} 2024-02-20T00:24:58.305 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expand map: {default=daemon} 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expand map: {default=debug} 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:24:58.035+0000 7ff4d6f47700 20 expand_channel_meta expanded map: {default=debug} 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 bash[38059]: f39385970b32eea6c0e6d61fc4d01769b9dffa1c0c8ce381da9a56b884a57f8e 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 systemd[1]: Started Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715. 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:24:58.051404+0000 mon.smithi087 (mon.0) 1 : cluster [INF] mon.smithi087 is new leader, mons smithi087 in quorum (ranks 0) 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:24:58.051463+0000 mon.smithi087 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi087=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:24:58.051494+0000 mon.smithi087 (mon.0) 3 : cluster [DBG] fsmap 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:24:58.051517+0000 mon.smithi087 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T00:24:58.306 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:24:58.051860+0000 mon.smithi087 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.576+0000 7fd18dbdd700 1 Processor -- start 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18dbdd700 1 -- start start 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18dbdd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18dbdd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880fc6e0 con 0x7fd1880703d0 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18cbdb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18cbdb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52324/0 (socket says 172.21.15.87:52324) 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.577+0000 7fd18cbdb700 1 -- 172.21.15.87:0/1353116488 learned_addr learned my addr 172.21.15.87:0/1353116488 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:58.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18cbdb700 1 -- 172.21.15.87:0/1353116488 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1881011e0 con 0x7fd1880703d0 2024-02-20T00:24:58.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18cbdb700 1 --2- 172.21.15.87:0/1353116488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd17c00b5f0 tx=0x7fd17c006ee0).ready entity=mon.0 client_cookie=ab02fb4e75788ccc server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:58.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd1877fe700 1 -- 172.21.15.87:0/1353116488 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c00d600 con 0x7fd1880703d0 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd1877fe700 1 -- 172.21.15.87:0/1353116488 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd17c00dbf0 con 0x7fd1880703d0 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd1877fe700 1 -- 172.21.15.87:0/1353116488 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c011cb0 con 0x7fd1880703d0 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 -- 172.21.15.87:0/1353116488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 msgr2=0x7fd188100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 --2- 172.21.15.87:0/1353116488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fd17c00b5f0 tx=0x7fd17c006ee0).stop 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 -- 172.21.15.87:0/1353116488 shutdown_connections 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 --2- 172.21.15.87:0/1353116488 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd188100cf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 -- 172.21.15.87:0/1353116488 >> 172.21.15.87:0/1353116488 conn(0x7fd1880f6390 msgr2=0x7fd1880f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.578+0000 7fd18dbdd700 1 -- 172.21.15.87:0/1353116488 shutdown_connections 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.579+0000 7fd18dbdd700 1 -- 172.21.15.87:0/1353116488 wait complete. 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.579+0000 7fd18dbdd700 1 Processor -- start 2024-02-20T00:24:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.579+0000 7fd18dbdd700 1 -- start start 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.579+0000 7fd18dbdd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.579+0000 7fd18dbdd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18810d090 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18cbdb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18cbdb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52334/0 (socket says 172.21.15.87:52334) 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18cbdb700 1 -- 172.21.15.87:0/749514495 learned_addr learned my addr 172.21.15.87:0/749514495 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18cbdb700 1 -- 172.21.15.87:0/749514495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c00b200 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18cbdb700 1 --2- 172.21.15.87:0/749514495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd17c006a20 tx=0x7fd17c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c01b570 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd17c01bb60 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c01aa20 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18810d290 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18810d720 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd17c00f070 con 0x7fd1880703d0 2024-02-20T00:24:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.580+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd17c025cd0 con 0x7fd1880703d0 2024-02-20T00:24:58.586 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.581+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188045ad0 con 0x7fd1880703d0 2024-02-20T00:24:58.586 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.583+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd17c016070 con 0x7fd1880703d0 2024-02-20T00:24:58.651 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.648+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd188101e60 con 0x7fd1880703d0 2024-02-20T00:24:58.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd1857fa700 1 -- 172.21.15.87:0/749514495 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd17c01a370 con 0x7fd1880703d0 2024-02-20T00:24:58.654 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 msgr2=0x7fd18810cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 --2- 172.21.15.87:0/749514495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd17c006a20 tx=0x7fd17c006ee0).stop 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 shutdown_connections 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 --2- 172.21.15.87:0/749514495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1880703d0 0x7fd18810cba0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 >> 172.21.15.87:0/749514495 conn(0x7fd1880f6390 msgr2=0x7fd1880f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.651+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 shutdown_connections 2024-02-20T00:24:58.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:24:58.652+0000 7fd18dbdd700 1 -- 172.21.15.87:0/749514495 wait complete. 2024-02-20T00:24:59.219 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-20T00:24:59.219 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:24:59.219 INFO:teuthology.orchestra.run.smithi087.stderr:Creating mgr... 2024-02-20T00:24:59.219 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying port 9283 ... 2024-02-20T00:24:59.221 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl daemon-reload 2024-02-20T00:24:59.315 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo 2024-02-20T00:24:59.337 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl reset-failed ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo 2024-02-20T00:24:59.342 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to reset failed state of unit ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo.service: Unit ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo.service not loaded. 2024-02-20T00:24:59.342 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl enable ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo 2024-02-20T00:24:59.348 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715.target.wants/ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo.service → /etc/systemd/system/ceph-6575da04-cf86-11ee-95bb-87774f69a715@.service. 2024-02-20T00:24:59.432 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@mgr.smithi087.iydsqo 2024-02-20T00:24:59.994 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:24:59 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:24:58.652748+0000 mon.smithi087 (mon.0) 6 : audit [INF] from='client.? 172.21.15.87:0/749514495' entity='client.admin' 2024-02-20T00:24:59.998 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:25:00.006 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-20T00:25:00.007 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:25:00.014 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:25:00.015 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-20T00:25:00.015 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T00:25:00.015 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-20T00:25:00.015 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:25:00.022 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-20T00:25:00.023 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:25:00.030 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:25:00.031 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-20T00:25:00.031 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T00:25:00.031 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr to start... 2024-02-20T00:25:00.031 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr... 2024-02-20T00:25:00.031 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:25:00.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.630+0000 7f5b4d420700 1 Processor -- start 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b4d420700 1 -- start start 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b4d420700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b4d420700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480fb450 con 0x7f5b480fab40 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52368/0 (socket says 172.21.15.87:52368) 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.631+0000 7f5b46ffd700 1 -- 172.21.15.87:0/3286191690 learned_addr learned my addr 172.21.15.87:0/3286191690 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b46ffd700 1 -- 172.21.15.87:0/3286191690 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b480fb590 con 0x7f5b480fab40 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b46ffd700 1 --2- 172.21.15.87:0/3286191690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5b3800b5f0 tx=0x7f5b38006ee0).ready entity=mon.0 client_cookie=270167c97e84c80e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b45ffb700 1 -- 172.21.15.87:0/3286191690 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3800d600 con 0x7f5b480fab40 2024-02-20T00:25:00.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b45ffb700 1 -- 172.21.15.87:0/3286191690 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5b3800dbf0 con 0x7f5b480fab40 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b45ffb700 1 -- 172.21.15.87:0/3286191690 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b38011cb0 con 0x7f5b480fab40 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 -- 172.21.15.87:0/3286191690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 msgr2=0x7f5b480faf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 --2- 172.21.15.87:0/3286191690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5b3800b5f0 tx=0x7f5b38006ee0).stop 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 -- 172.21.15.87:0/3286191690 shutdown_connections 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 --2- 172.21.15.87:0/3286191690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b480faf60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 -- 172.21.15.87:0/3286191690 >> 172.21.15.87:0/3286191690 conn(0x7f5b480f63c0 msgr2=0x7f5b480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.632+0000 7f5b4d420700 1 -- 172.21.15.87:0/3286191690 shutdown_connections 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.633+0000 7f5b4d420700 1 -- 172.21.15.87:0/3286191690 wait complete. 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.633+0000 7f5b4d420700 1 Processor -- start 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.633+0000 7f5b4d420700 1 -- start start 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.633+0000 7f5b4d420700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.633+0000 7f5b4d420700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4810aec0 con 0x7f5b480fab40 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:00.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52380/0 (socket says 172.21.15.87:52380) 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b46ffd700 1 -- 172.21.15.87:0/3591125608 learned_addr learned my addr 172.21.15.87:0/3591125608 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b46ffd700 1 -- 172.21.15.87:0/3591125608 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3800b200 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b46ffd700 1 --2- 172.21.15.87:0/3591125608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5b38002f40 tx=0x7f5b38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3801b410 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4810b0c0 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5b3801ba00 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3801a900 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4810b550 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5b3800f070 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.634+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5b38025c80 con 0x7f5b480fab40 2024-02-20T00:25:00.640 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.635+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b48045ad0 con 0x7f5b480fab40 2024-02-20T00:25:00.641 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.637+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5b38016070 con 0x7f5b480fab40 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.707+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5b480fc910 con 0x7f5b480fab40 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.707+0000 7f5b2f7fe700 1 -- 172.21.15.87:0/3591125608 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5b38025460 con 0x7f5b480fab40 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "6575da04-cf86-11ee-95bb-87774f69a715", 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "smithi087" 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:25:00.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:25:00.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": false, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:24:52.707649+0000", 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 msgr2=0x7f5b4810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 --2- 172.21.15.87:0/3591125608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5b38002f40 tx=0x7f5b38006ee0).stop 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 shutdown_connections 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 --2- 172.21.15.87:0/3591125608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5b480fab40 0x7f5b4810a9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 >> 172.21.15.87:0/3591125608 conn(0x7f5b480f63c0 msgr2=0x7f5b48070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 shutdown_connections 2024-02-20T00:25:00.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:00.708+0000 7f5b4d420700 1 -- 172.21.15.87:0/3591125608 wait complete. 2024-02-20T00:25:01.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:00.709531+0000 mon.smithi087 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.87:0/3591125608' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:25:01.304 INFO:teuthology.orchestra.run.smithi087.stderr:mgr not available, waiting (1/15)... 2024-02-20T00:25:03.306 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.792+0000 7f0b952a6700 1 Processor -- start 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.792+0000 7f0b952a6700 1 -- start start 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b952a6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b952a6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900ff050 con 0x7f0b90070440 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52394/0 (socket says 172.21.15.87:52394) 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8effd700 1 -- 172.21.15.87:0/1546846681 learned_addr learned my addr 172.21.15.87:0/1546846681 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8effd700 1 -- 172.21.15.87:0/1546846681 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900ff190 con 0x7f0b90070440 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8effd700 1 --2- 172.21.15.87:0/1546846681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0b8000b5f0 tx=0x7f0b80006ee0).ready entity=mon.0 client_cookie=ee0ecc1362b8be76 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:03.797 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8dffb700 1 -- 172.21.15.87:0/1546846681 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b8000d600 con 0x7f0b90070440 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8dffb700 1 -- 172.21.15.87:0/1546846681 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0b8000dbf0 con 0x7f0b90070440 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.793+0000 7f0b8dffb700 1 -- 172.21.15.87:0/1546846681 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b80011cb0 con 0x7f0b90070440 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 -- 172.21.15.87:0/1546846681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 msgr2=0x7f0b900feb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 --2- 172.21.15.87:0/1546846681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0b8000b5f0 tx=0x7f0b80006ee0).stop 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 -- 172.21.15.87:0/1546846681 shutdown_connections 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 --2- 172.21.15.87:0/1546846681 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b900feb60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 -- 172.21.15.87:0/1546846681 >> 172.21.15.87:0/1546846681 conn(0x7f0b900f65c0 msgr2=0x7f0b900f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 -- 172.21.15.87:0/1546846681 shutdown_connections 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.794+0000 7f0b952a6700 1 -- 172.21.15.87:0/1546846681 wait complete. 2024-02-20T00:25:03.799 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b952a6700 1 Processor -- start 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b952a6700 1 -- start start 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b952a6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b952a6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9010b0f0 con 0x7f0b90070440 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52402/0 (socket says 172.21.15.87:52402) 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b8effd700 1 -- 172.21.15.87:0/2731840016 learned_addr learned my addr 172.21.15.87:0/2731840016 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.795+0000 7f0b8effd700 1 -- 172.21.15.87:0/2731840016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8000b200 con 0x7f0b90070440 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b8effd700 1 --2- 172.21.15.87:0/2731840016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b8000df90 tx=0x7f0b80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b8001b410 con 0x7f0b90070440 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0b8001ba00 con 0x7f0b90070440 2024-02-20T00:25:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9010b2f0 con 0x7f0b90070440 2024-02-20T00:25:03.801 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9010b780 con 0x7f0b90070440 2024-02-20T00:25:03.801 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.796+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b8001aa30 con 0x7f0b90070440 2024-02-20T00:25:03.801 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.797+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0b8000f070 con 0x7f0b90070440 2024-02-20T00:25:03.801 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.797+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0b800258c0 con 0x7f0b90070440 2024-02-20T00:25:03.801 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.797+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b70004fa0 con 0x7f0b90070440 2024-02-20T00:25:03.802 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.799+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0b80016070 con 0x7f0b90070440 2024-02-20T00:25:03.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.868+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0b70005670 con 0x7f0b90070440 2024-02-20T00:25:03.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.868+0000 7f0b777fe700 1 -- 172.21.15.87:0/2731840016 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0b8001a370 con 0x7f0b90070440 2024-02-20T00:25:03.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:03.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "6575da04-cf86-11ee-95bb-87774f69a715", 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "smithi087" 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-20T00:25:03.878 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": false, 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:03.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:24:52.707649+0000", 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.869+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 msgr2=0x7f0b9010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.869+0000 7f0b952a6700 1 --2- 172.21.15.87:0/2731840016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b8000df90 tx=0x7f0b80006ee0).stop 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.869+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 shutdown_connections 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.869+0000 7f0b952a6700 1 --2- 172.21.15.87:0/2731840016 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b90070440 0x7f0b9010ac00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.869+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 >> 172.21.15.87:0/2731840016 conn(0x7f0b900f65c0 msgr2=0x7f0b900f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:03.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.870+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 shutdown_connections 2024-02-20T00:25:03.883 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:03.870+0000 7f0b952a6700 1 -- 172.21.15.87:0/2731840016 wait complete. 2024-02-20T00:25:04.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:03.870166+0000 mon.smithi087 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.87:0/2731840016' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:25:04.368 INFO:teuthology.orchestra.run.smithi087.stderr:mgr not available, waiting (2/15)... 2024-02-20T00:25:05.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:04.872558+0000 mon.smithi087 (mon.0) 9 : cluster [INF] Activating manager daemon smithi087.iydsqo 2024-02-20T00:25:05.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:04.874897+0000 mon.smithi087 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi087.iydsqo(active, starting, since 0.00242235s) 2024-02-20T00:25:05.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.875736+0000 mon.smithi087 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:25:05.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.875912+0000 mon.smithi087 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:25:05.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.876047+0000 mon.smithi087 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.876233+0000 mon.smithi087 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata", "id": "smithi087"}]: dispatch 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.876444+0000 mon.smithi087 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mgr metadata", "who": "smithi087.iydsqo", "id": "smithi087.iydsqo"}]: dispatch 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:04.878039+0000 mon.smithi087 (mon.0) 16 : cluster [INF] Manager daemon smithi087.iydsqo is now available 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.885738+0000 mon.smithi087 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.889502+0000 mon.smithi087 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.890571+0000 mon.smithi087 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.895443+0000 mon.smithi087 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:05.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:04.897643+0000 mon.smithi087 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.87:0/1874573781' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:06.371 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:25:07.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:05.878328+0000 mon.smithi087 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi087.iydsqo(active, since 1.00585s) 2024-02-20T00:25:08.286 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.944+0000 7fad218e9700 1 Processor -- start 2024-02-20T00:25:08.286 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.944+0000 7fad218e9700 1 -- start start 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.944+0000 7fad218e9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.944+0000 7fad218e9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c072920 con 0x7fad1c072500 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52456/0 (socket says 172.21.15.87:52456) 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad1affd700 1 -- 172.21.15.87:0/1641601938 learned_addr learned my addr 172.21.15.87:0/1641601938 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad1affd700 1 -- 172.21.15.87:0/1641601938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c070f90 con 0x7fad1c072500 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad1affd700 1 --2- 172.21.15.87:0/1641601938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fad1000b5f0 tx=0x7fad10006ee0).ready entity=mon.0 client_cookie=ba448fbbed618c99 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad19ffb700 1 -- 172.21.15.87:0/1641601938 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad1000d600 con 0x7fad1c072500 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad19ffb700 1 -- 172.21.15.87:0/1641601938 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fad1000dbf0 con 0x7fad1c072500 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.945+0000 7fad19ffb700 1 -- 172.21.15.87:0/1641601938 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad10011cb0 con 0x7fad1c072500 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- 172.21.15.87:0/1641601938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 msgr2=0x7fad1c070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 --2- 172.21.15.87:0/1641601938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fad1000b5f0 tx=0x7fad10006ee0).stop 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- 172.21.15.87:0/1641601938 shutdown_connections 2024-02-20T00:25:08.287 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 --2- 172.21.15.87:0/1641601938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c070aa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- 172.21.15.87:0/1641601938 >> 172.21.15.87:0/1641601938 conn(0x7fad1c0f76a0 msgr2=0x7fad1c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- 172.21.15.87:0/1641601938 shutdown_connections 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- 172.21.15.87:0/1641601938 wait complete. 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 Processor -- start 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.946+0000 7fad218e9700 1 -- start start 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad218e9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad218e9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c10d300 con 0x7fad1c072500 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52472/0 (socket says 172.21.15.87:52472) 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad1affd700 1 -- 172.21.15.87:0/3747034758 learned_addr learned my addr 172.21.15.87:0/3747034758 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad1affd700 1 -- 172.21.15.87:0/3747034758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1000b200 con 0x7fad1c072500 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad1affd700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fad10006800 tx=0x7fad10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:08.288 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad1001b570 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c10d500 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fad1001bb60 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad1001a940 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.947+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c10e110 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.948+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fad1000f070 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.948+0000 7fad037fe700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 0x7fad08032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.948+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fad100413e0 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.948+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c059100 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.948+0000 7fad1a7fc700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 0x7fad08032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.949+0000 7fad1a7fc700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 0x7fad08032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fad0c0096a0 tx=0x7fad0c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:07.953+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad10016070 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.158+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fad1c102140 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.159+0000 7fad037fe700 1 -- 172.21.15.87:0/3747034758 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fad1001b6d0 con 0x7fad1c072500 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 msgr2=0x7fad08032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:08.289 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 0x7fad08032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fad0c0096a0 tx=0x7fad0c006b40).stop 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 msgr2=0x7fad1c10ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fad10006800 tx=0x7fad10006ee0).stop 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 shutdown_connections 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fad0802fdc0 0x7fad08032280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 --2- 172.21.15.87:0/3747034758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad1c072500 0x7fad1c10ce10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 >> 172.21.15.87:0/3747034758 conn(0x7fad1c0f76a0 msgr2=0x7fad1c0f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 shutdown_connections 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.160+0000 7fad218e9700 1 -- 172.21.15.87:0/3747034758 wait complete. 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "6575da04-cf86-11ee-95bb-87774f69a715", 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:25:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "smithi087" 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:08.291 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.292 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.293 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:24:52.707649+0000", 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:25:08.294 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-20T00:25:08.295 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:25:08.295 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:09.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:07.878481+0000 mon.smithi087 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi087.iydsqo(active, since 3s) 2024-02-20T00:25:09.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:08.159628+0000 mon.smithi087 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.87:0/3747034758' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:25:12.532 INFO:teuthology.orchestra.run.smithi087.stderr:mgr is available 2024-02-20T00:25:12.532 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling cephadm module... 2024-02-20T00:25:12.532 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-20T00:25:16.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:16.216020+0000 mon.smithi087 (mon.0) 25 : audit [INF] from='client.? 172.21.15.87:0/3191657342' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.020+0000 7f7bd3cb8700 1 Processor -- start 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.020+0000 7f7bd3cb8700 1 -- start start 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.020+0000 7f7bd3cb8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.020+0000 7f7bd3cb8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc072940 con 0x7f7bcc072520 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd1a54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:16.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd1a54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43196/0 (socket says 172.21.15.87:43196) 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd1a54700 1 -- 172.21.15.87:0/1840238519 learned_addr learned my addr 172.21.15.87:0/1840238519 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd1a54700 1 -- 172.21.15.87:0/1840238519 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc070fb0 con 0x7f7bcc072520 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd1a54700 1 --2- 172.21.15.87:0/1840238519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7bbc00b5f0 tx=0x7f7bbc006ee0).ready entity=mon.0 client_cookie=56727de91387ff9f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd0a52700 1 -- 172.21.15.87:0/1840238519 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc00d600 con 0x7f7bcc072520 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd0a52700 1 -- 172.21.15.87:0/1840238519 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7bbc00dbf0 con 0x7f7bcc072520 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.021+0000 7f7bd0a52700 1 -- 172.21.15.87:0/1840238519 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc011cb0 con 0x7f7bcc072520 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/1840238519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 msgr2=0x7f7bcc070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/1840238519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7bbc00b5f0 tx=0x7f7bbc006ee0).stop 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/1840238519 shutdown_connections 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/1840238519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc070ac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/1840238519 >> 172.21.15.87:0/1840238519 conn(0x7f7bcc0f76a0 msgr2=0x7f7bcc0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/1840238519 shutdown_connections 2024-02-20T00:25:16.615 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/1840238519 wait complete. 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 Processor -- start 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.022+0000 7f7bd3cb8700 1 -- start start 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd3cb8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd3cb8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc10d2f0 con 0x7f7bcc072520 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd1a54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd1a54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43200/0 (socket says 172.21.15.87:43200) 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd1a54700 1 -- 172.21.15.87:0/3191657342 learned_addr learned my addr 172.21.15.87:0/3191657342 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd1a54700 1 -- 172.21.15.87:0/3191657342 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc00b200 con 0x7f7bcc072520 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd1a54700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7bbc006800 tx=0x7f7bbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc01b570 con 0x7f7bcc072520 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7bbc01bb60 con 0x7f7bcc072520 2024-02-20T00:25:16.616 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc10d4f0 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc01a9f0 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.023+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc10e100 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f7bbc00f070 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bba7fc700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 0x7f7bb4032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7bbc041480 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc059100 con 0x7f7bcc072520 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bd1253700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 0x7f7bb4032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.024+0000 7f7bd1253700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 0x7f7bb4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7bc80096a0 tx=0x7f7bc8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:16.617 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.028+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bbc022310 con 0x7f7bcc072520 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.214+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7bcc045ad0 con 0x7f7bcc072520 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.267+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7bbc01bcd0 con 0x7f7bcc072520 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.268+0000 7f7bba7fc700 1 -- 172.21.15.87:0/3191657342 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7bbc0239f0 con 0x7f7bcc072520 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 msgr2=0x7f7bb4032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 0x7f7bb4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7bc80096a0 tx=0x7f7bc8006b40).stop 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 msgr2=0x7f7bcc10ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7bbc006800 tx=0x7f7bbc006ee0).stop 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 shutdown_connections 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f7bb402fdc0 0x7f7bb4032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 --2- 172.21.15.87:0/3191657342 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bcc072520 0x7f7bcc10ce00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 >> 172.21.15.87:0/3191657342 conn(0x7f7bcc0f76a0 msgr2=0x7f7bcc0f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 shutdown_connections 2024-02-20T00:25:16.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:16.270+0000 7f7bd3cb8700 1 -- 172.21.15.87:0/3191657342 wait complete. 2024-02-20T00:25:17.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:17 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:16.268935+0000 mon.smithi087 (mon.0) 26 : audit [INF] from='client.? 172.21.15.87:0/3191657342' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T00:25:17.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:17 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:16.268977+0000 mon.smithi087 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi087.iydsqo(active, since 11s) 2024-02-20T00:25:18.566 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.114+0000 7f790e7ea700 1 Processor -- start 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.114+0000 7f790e7ea700 1 -- start start 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.114+0000 7f790e7ea700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.114+0000 7f790e7ea700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79080fc980 con 0x7f79080703b0 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7907fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7907fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50474/0 (socket says 172.21.15.87:50474) 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7907fff700 1 -- 172.21.15.87:0/563174174 learned_addr learned my addr 172.21.15.87:0/563174174 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7907fff700 1 -- 172.21.15.87:0/563174174 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7908101480 con 0x7f79080703b0 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7907fff700 1 --2- 172.21.15.87:0/563174174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f78f800b5f0 tx=0x7f78f8006ee0).ready entity=mon.0 client_cookie=5069817d3bc3be36 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7906ffd700 1 -- 172.21.15.87:0/563174174 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f800d600 con 0x7f79080703b0 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7906ffd700 1 -- 172.21.15.87:0/563174174 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f78f800dbf0 con 0x7f79080703b0 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.115+0000 7f7906ffd700 1 -- 172.21.15.87:0/563174174 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f8011cb0 con 0x7f79080703b0 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 -- 172.21.15.87:0/563174174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 msgr2=0x7f7908100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 --2- 172.21.15.87:0/563174174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f78f800b5f0 tx=0x7f78f8006ee0).stop 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 -- 172.21.15.87:0/563174174 shutdown_connections 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 --2- 172.21.15.87:0/563174174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f7908100f90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 -- 172.21.15.87:0/563174174 >> 172.21.15.87:0/563174174 conn(0x7f79080f65b0 msgr2=0x7f79080f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 -- 172.21.15.87:0/563174174 shutdown_connections 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.116+0000 7f790e7ea700 1 -- 172.21.15.87:0/563174174 wait complete. 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.117+0000 7f790e7ea700 1 Processor -- start 2024-02-20T00:25:19.120 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.117+0000 7f790e7ea700 1 -- start start 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.117+0000 7f790e7ea700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f790e7ea700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790810d2c0 con 0x7f79080703b0 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7907fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7907fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50476/0 (socket says 172.21.15.87:50476) 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7907fff700 1 -- 172.21.15.87:0/3476168713 learned_addr learned my addr 172.21.15.87:0/3476168713 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7907fff700 1 -- 172.21.15.87:0/3476168713 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f800b200 con 0x7f79080703b0 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7907fff700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f78f8006800 tx=0x7f78f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:19.122 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f801b570 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790810d4c0 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f78f801bb60 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f801a9f0 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.118+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790810e100 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.119+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f78f800f070 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.119+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790810d950 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.119+0000 7f7904ff9700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 0x7f78f40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.120+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f78f80414d0 con 0x7f79080703b0 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.120+0000 7f79077fe700 1 -- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 msgr2=0x7f78f40322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/446918819 2024-02-20T00:25:19.123 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.120+0000 7f79077fe700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 0x7f78f40322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:25:19.128 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.126+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78f8010360 con 0x7f79080703b0 2024-02-20T00:25:19.323 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.321+0000 7f79077fe700 1 -- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 msgr2=0x7f78f40322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/446918819 2024-02-20T00:25:19.323 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.321+0000 7f79077fe700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 0x7f78f40322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.333+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7908102210 con 0x7f79080703b0 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.333+0000 7f7904ff9700 1 -- 172.21.15.87:0/3476168713 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f78f8016070 con 0x7f79080703b0 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "active_name": "smithi087.iydsqo", 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T00:25:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 msgr2=0x7f78f40322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 0x7f78f40322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 msgr2=0x7f790810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f78f8006800 tx=0x7f78f8006ee0).stop 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 shutdown_connections 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7f78f402fe10 0x7f78f40322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:19.338 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 --2- 172.21.15.87:0/3476168713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f79080703b0 0x7f790810cdd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:19.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 >> 172.21.15.87:0/3476168713 conn(0x7f79080f65b0 msgr2=0x7f79080f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:19.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 shutdown_connections 2024-02-20T00:25:19.339 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:19.335+0000 7f790e7ea700 1 -- 172.21.15.87:0/3476168713 wait complete. 2024-02-20T00:25:19.677 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:19 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:19.334442+0000 mon.smithi087 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.87:0/3476168713' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T00:25:19.914 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for the mgr to restart... 2024-02-20T00:25:19.915 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr epoch 5... 2024-02-20T00:25:19.915 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-20T00:25:20.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.459+0000 7fed454b8700 1 Processor -- start 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.459+0000 7fed454b8700 1 -- start start 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed454b8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed454b8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed40070770 con 0x7fed40070350 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50492/0 (socket says 172.21.15.87:50492) 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3effd700 1 -- 172.21.15.87:0/2332582266 learned_addr learned my addr 172.21.15.87:0/2332582266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3effd700 1 -- 172.21.15.87:0/2332582266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400ff0e0 con 0x7fed40070350 2024-02-20T00:25:20.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3effd700 1 --2- 172.21.15.87:0/2332582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fed3000b5f0 tx=0x7fed30006ee0).ready entity=mon.0 client_cookie=979e7e4a6e73081a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:20.477 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3dffb700 1 -- 172.21.15.87:0/2332582266 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3000d600 con 0x7fed40070350 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3dffb700 1 -- 172.21.15.87:0/2332582266 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed3000dbf0 con 0x7fed40070350 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.460+0000 7fed3dffb700 1 -- 172.21.15.87:0/2332582266 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed30011cb0 con 0x7fed40070350 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 -- 172.21.15.87:0/2332582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 msgr2=0x7fed400febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 --2- 172.21.15.87:0/2332582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fed3000b5f0 tx=0x7fed30006ee0).stop 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 -- 172.21.15.87:0/2332582266 shutdown_connections 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 --2- 172.21.15.87:0/2332582266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400febf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 -- 172.21.15.87:0/2332582266 >> 172.21.15.87:0/2332582266 conn(0x7fed400f65b0 msgr2=0x7fed400f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 -- 172.21.15.87:0/2332582266 shutdown_connections 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.461+0000 7fed454b8700 1 -- 172.21.15.87:0/2332582266 wait complete. 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 Processor -- start 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 -- start start 2024-02-20T00:25:20.478 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed400faaf0 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50498/0 (socket says 172.21.15.87:50498) 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed3effd700 1 -- 172.21.15.87:0/2439842194 learned_addr learned my addr 172.21.15.87:0/2439842194 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed3effd700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3000b200 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed3effd700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fed30006800 tx=0x7fed30006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3001b570 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed3001bb60 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed400fac90 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3001a9f0 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.462+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed400fb120 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fed3000f070 con 0x7fed40070350 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed277fe700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed40059100 con 0x7fed2c02fe10 2024-02-20T00:25:20.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fed30041450 con 0x7fed40070350 2024-02-20T00:25:20.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed3e7fc700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 msgr2=0x7fed2c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/446918819 2024-02-20T00:25:20.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.463+0000 7fed3e7fc700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:25:20.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.664+0000 7fed3e7fc700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 msgr2=0x7fed2c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/446918819 2024-02-20T00:25:20.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:20.664+0000 7fed3e7fc700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:25:21.067 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:21.065+0000 7fed3e7fc700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 msgr2=0x7fed2c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/446918819 2024-02-20T00:25:21.067 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:21.065+0000 7fed3e7fc700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:25:21.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:21.259+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fed3001bcd0 con 0x7fed40070350 2024-02-20T00:25:21.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:21.260+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 msgr2=0x7fed2c0322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:25:21.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:21.260+0000 7fed277fe700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:21.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:21 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:21.207563+0000 mon.smithi087 (mon.0) 29 : cluster [INF] Active manager daemon smithi087.iydsqo restarted 2024-02-20T00:25:21.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:21 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:21.207931+0000 mon.smithi087 (mon.0) 30 : cluster [INF] Activating manager daemon smithi087.iydsqo 2024-02-20T00:25:21.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:21 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:21.210225+0000 mon.smithi087 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-20T00:25:22.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.263+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fed30042a60 con 0x7fed40070350 2024-02-20T00:25:22.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.263+0000 7fed277fe700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 0x7fed2c035b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:22.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.263+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed40059100 con 0x7fed2c033730 2024-02-20T00:25:22.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.264+0000 7fed3e7fc700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 0x7fed2c035b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:22.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.264+0000 7fed3e7fc700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 0x7fed2c035b20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fed340036d0 tx=0x7fed34006c50).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:22.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.265+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fed40059100 con 0x7fed2c033730 2024-02-20T00:25:22.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.272+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fed40045ad0 con 0x7fed2c033730 2024-02-20T00:25:22.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed277fe700 1 -- 172.21.15.87:0/2439842194 <== mgr.14114 v2:172.21.15.87:6800/67270349 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fed40045ad0 con 0x7fed2c033730 2024-02-20T00:25:22.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:22.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-20T00:25:22.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T00:25:22.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 msgr2=0x7fed2c035b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 0x7fed2c035b20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fed340036d0 tx=0x7fed34006c50).stop 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 msgr2=0x7fed400fc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fed30006800 tx=0x7fed30006ee0).stop 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 shutdown_connections 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fed2c033730 0x7fed2c035b20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed40070350 0x7fed400fc670 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 --2- 172.21.15.87:0/2439842194 >> [v2:172.21.15.87:6800/446918819,v1:172.21.15.87:6801/446918819] conn(0x7fed2c02fe10 0x7fed2c0322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 >> 172.21.15.87:0/2439842194 conn(0x7fed400f65b0 msgr2=0x7fed400f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 shutdown_connections 2024-02-20T00:25:22.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:22.273+0000 7fed454b8700 1 -- 172.21.15.87:0/2439842194 wait complete. 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:21.261297+0000 mon.smithi087 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi087.iydsqo(active, starting, since 0.053439s) 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.263905+0000 mon.smithi087 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata", "id": "smithi087"}]: dispatch 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.264574+0000 mon.smithi087 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mgr metadata", "who": "smithi087.iydsqo", "id": "smithi087.iydsqo"}]: dispatch 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.265237+0000 mon.smithi087 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.265412+0000 mon.smithi087 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.265551+0000 mon.smithi087 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:25:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:21.279267+0000 mon.smithi087 (mon.0) 38 : cluster [INF] Manager daemon smithi087.iydsqo is now available 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.292785+0000 mon.smithi087 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.295466+0000 mon.smithi087 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.297989+0000 mon.smithi087 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.385089+0000 mon.smithi087 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.385747+0000 mon.smithi087 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.387489+0000 mon.smithi087 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.388741+0000 mon.smithi087 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.398245+0000 mon.smithi087 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:25:22.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:21.399325+0000 mon.smithi087 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:25:22.930 INFO:teuthology.orchestra.run.smithi087.stderr:mgr epoch 5 is available 2024-02-20T00:25:22.930 INFO:teuthology.orchestra.run.smithi087.stderr:Setting orchestrator backend to cephadm... 2024-02-20T00:25:22.931 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-20T00:25:23.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.461+0000 7ff74807c700 1 Processor -- start 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.461+0000 7ff74807c700 1 -- start start 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.461+0000 7ff74807c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.461+0000 7ff74807c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400fb6a0 con 0x7ff7400fad90 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.461+0000 7ff745e18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff745e18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50540/0 (socket says 172.21.15.87:50540) 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff745e18700 1 -- 172.21.15.87:0/3632654395 learned_addr learned my addr 172.21.15.87:0/3632654395 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff745e18700 1 -- 172.21.15.87:0/3632654395 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7400fb7e0 con 0x7ff7400fad90 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff745e18700 1 --2- 172.21.15.87:0/3632654395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff73000b5f0 tx=0x7ff730006ee0).ready entity=mon.0 client_cookie=57c684a0f5421a2c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff744e16700 1 -- 172.21.15.87:0/3632654395 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff73000d600 con 0x7ff7400fad90 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff744e16700 1 -- 172.21.15.87:0/3632654395 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff73000dbf0 con 0x7ff7400fad90 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.462+0000 7ff744e16700 1 -- 172.21.15.87:0/3632654395 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff730011cb0 con 0x7ff7400fad90 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- 172.21.15.87:0/3632654395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 msgr2=0x7ff7400fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:23.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 --2- 172.21.15.87:0/3632654395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff73000b5f0 tx=0x7ff730006ee0).stop 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- 172.21.15.87:0/3632654395 shutdown_connections 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 --2- 172.21.15.87:0/3632654395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff7400fb1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- 172.21.15.87:0/3632654395 >> 172.21.15.87:0/3632654395 conn(0x7ff7400f6530 msgr2=0x7ff7400f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- 172.21.15.87:0/3632654395 shutdown_connections 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- 172.21.15.87:0/3632654395 wait complete. 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 Processor -- start 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- start start 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:23.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.463+0000 7ff74807c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740070470 con 0x7ff7400fad90 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff745e18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff745e18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50552/0 (socket says 172.21.15.87:50552) 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff745e18700 1 -- 172.21.15.87:0/2149900169 learned_addr learned my addr 172.21.15.87:0/2149900169 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff745e18700 1 -- 172.21.15.87:0/2149900169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73000b200 con 0x7ff7400fad90 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff745e18700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff730011ea0 tx=0x7ff730006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:23.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff73001b570 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff73001bb60 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff73001a8d0 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740070670 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.464+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74010deb0 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.465+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ff73000f070 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.465+0000 7ff72e7fc700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 0x7ff728032220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.465+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff740045ad0 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.465+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff7300413c0 con 0x7ff7400fad90 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.465+0000 7ff745617700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 0x7ff728032220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:23.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.466+0000 7ff745617700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 0x7ff728032220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff73c0096a0 tx=0x7ff73c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:23.471 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.470+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff730016070 con 0x7ff7400fad90 2024-02-20T00:25:23.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:23 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:22.265719+0000 mgr.smithi087.iydsqo (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T00:25:23.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:23 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:22.266842+0000 mon.smithi087 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi087.iydsqo(active, since 1.05898s) 2024-02-20T00:25:23.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:23 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:22.273674+0000 mgr.smithi087.iydsqo (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T00:25:23.650 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.648+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff740057f10 con 0x7ff72802fd60 2024-02-20T00:25:23.655 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.654+0000 7ff72e7fc700 1 -- 172.21.15.87:0/2149900169 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff740057f10 con 0x7ff72802fd60 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 msgr2=0x7ff728032220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 0x7ff728032220 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff73c0096a0 tx=0x7ff73c006b40).stop 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 msgr2=0x7ff74010d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff730011ea0 tx=0x7ff730006ee0).stop 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 shutdown_connections 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7ff72802fd60 0x7ff728032220 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:23.658 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 --2- 172.21.15.87:0/2149900169 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7400fad90 0x7ff74010d800 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:23.659 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 >> 172.21.15.87:0/2149900169 conn(0x7ff7400f6530 msgr2=0x7ff7400f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:23.659 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 shutdown_connections 2024-02-20T00:25:23.659 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:23.655+0000 7ff74807c700 1 -- 172.21.15.87:0/2149900169 wait complete. 2024-02-20T00:25:24.303 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-20T00:25:24.817 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.812+0000 7f34a0565700 1 Processor -- start 2024-02-20T00:25:24.817 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.812+0000 7f34a0565700 1 -- start start 2024-02-20T00:25:24.817 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.812+0000 7f34a0565700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:24.817 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f34a0565700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980fef80 con 0x7f34980703f0 2024-02-20T00:25:24.817 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349e301700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349e301700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50568/0 (socket says 172.21.15.87:50568) 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349e301700 1 -- 172.21.15.87:0/3262390266 learned_addr learned my addr 172.21.15.87:0/3262390266 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349e301700 1 -- 172.21.15.87:0/3262390266 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980ff0c0 con 0x7f34980703f0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349e301700 1 --2- 172.21.15.87:0/3262390266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f348800b5f0 tx=0x7f3488006ee0).ready entity=mon.0 client_cookie=7f9991ed78128248 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349d2ff700 1 -- 172.21.15.87:0/3262390266 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f348800d600 con 0x7f34980703f0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349d2ff700 1 -- 172.21.15.87:0/3262390266 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f348800dbf0 con 0x7f34980703f0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.813+0000 7f349d2ff700 1 -- 172.21.15.87:0/3262390266 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3488011ce0 con 0x7f34980703f0 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 -- 172.21.15.87:0/3262390266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 msgr2=0x7f34980fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 --2- 172.21.15.87:0/3262390266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f348800b5f0 tx=0x7f3488006ee0).stop 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 -- 172.21.15.87:0/3262390266 shutdown_connections 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 --2- 172.21.15.87:0/3262390266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f34980fea90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 -- 172.21.15.87:0/3262390266 >> 172.21.15.87:0/3262390266 conn(0x7f34980f6530 msgr2=0x7f34980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 -- 172.21.15.87:0/3262390266 shutdown_connections 2024-02-20T00:25:24.818 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 -- 172.21.15.87:0/3262390266 wait complete. 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.814+0000 7f34a0565700 1 Processor -- start 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f34a0565700 1 -- start start 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f34a0565700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f34a0565700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349810b080 con 0x7f34980703f0 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f349e301700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:24.819 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f349e301700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50570/0 (socket says 172.21.15.87:50570) 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f349e301700 1 -- 172.21.15.87:0/1351815998 learned_addr learned my addr 172.21.15.87:0/1351815998 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f349e301700 1 -- 172.21.15.87:0/1351815998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348800b200 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f349e301700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f34880115c0 tx=0x7f3488006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f348801b410 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f348801ba00 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349810b280 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f348801aaa0 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.815+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349810b710 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.816+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f348800f070 con 0x7f34980703f0 2024-02-20T00:25:24.820 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.816+0000 7f3486ffd700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 0x7f34800322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:24.821 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.816+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3488041110 con 0x7f34980703f0 2024-02-20T00:25:24.821 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.816+0000 7f349db00700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 0x7f34800322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:24.821 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.817+0000 7f349db00700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 0x7f34800322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f34940096a0 tx=0x7f3494006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:24.821 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.817+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3498045ad0 con 0x7f34980703f0 2024-02-20T00:25:24.823 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.821+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3488016070 con 0x7f34980703f0 2024-02-20T00:25:24.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.971+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3498104c00 con 0x7f348002fe10 2024-02-20T00:25:24.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.972+0000 7f3486ffd700 1 -- 172.21.15.87:0/1351815998 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3498104c00 con 0x7f348002fe10 2024-02-20T00:25:24.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: value unchanged 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 msgr2=0x7f34800322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 0x7f34800322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f34940096a0 tx=0x7f3494006b40).stop 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 msgr2=0x7f349810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f34880115c0 tx=0x7f3488006ee0).stop 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 shutdown_connections 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f348002fe10 0x7f34800322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 --2- 172.21.15.87:0/1351815998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34980703f0 0x7f349810ab90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 >> 172.21.15.87:0/1351815998 conn(0x7f34980f6530 msgr2=0x7f34980f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 shutdown_connections 2024-02-20T00:25:24.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:24.973+0000 7f34a0565700 1 -- 172.21.15.87:0/1351815998 wait complete. 2024-02-20T00:25:24.999 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:23.649989+0000 mgr.smithi087.iydsqo (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:24.999 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:23.653884+0000 mon.smithi087 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:24.999 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:23.657295+0000 mon.smithi087 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:24.999 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:24.047652+0000 mon.smithi087 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi087.iydsqo(active, since 2s) 2024-02-20T00:25:25.475 INFO:teuthology.orchestra.run.smithi087.stderr:Generating ssh key... 2024-02-20T00:25:25.475 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-20T00:25:25.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:25 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:24.972689+0000 mgr.smithi087.iydsqo (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:25.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.942+0000 7f582c133700 1 Processor -- start 2024-02-20T00:25:25.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.942+0000 7f582c133700 1 -- start start 2024-02-20T00:25:25.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.942+0000 7f582c133700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:25.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.942+0000 7f582c133700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58240fef40 con 0x7f5824070430 2024-02-20T00:25:25.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5829ecf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5829ecf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50576/0 (socket says 172.21.15.87:50576) 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5829ecf700 1 -- 172.21.15.87:0/1192071180 learned_addr learned my addr 172.21.15.87:0/1192071180 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5829ecf700 1 -- 172.21.15.87:0/1192071180 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58240ff080 con 0x7f5824070430 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5829ecf700 1 --2- 172.21.15.87:0/1192071180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f581400b5f0 tx=0x7f5814006ee0).ready entity=mon.0 client_cookie=5e764f01d8a24412 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5828ecd700 1 -- 172.21.15.87:0/1192071180 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581400d600 con 0x7f5824070430 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5828ecd700 1 -- 172.21.15.87:0/1192071180 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f581400dbf0 con 0x7f5824070430 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.943+0000 7f5828ecd700 1 -- 172.21.15.87:0/1192071180 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5814011cb0 con 0x7f5824070430 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 -- 172.21.15.87:0/1192071180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 msgr2=0x7f58240fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 --2- 172.21.15.87:0/1192071180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f581400b5f0 tx=0x7f5814006ee0).stop 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 -- 172.21.15.87:0/1192071180 shutdown_connections 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 --2- 172.21.15.87:0/1192071180 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fea50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 -- 172.21.15.87:0/1192071180 >> 172.21.15.87:0/1192071180 conn(0x7f58240f6530 msgr2=0x7f58240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 -- 172.21.15.87:0/1192071180 shutdown_connections 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 -- 172.21.15.87:0/1192071180 wait complete. 2024-02-20T00:25:25.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.944+0000 7f582c133700 1 Processor -- start 2024-02-20T00:25:25.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f582c133700 1 -- start start 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f582c133700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f582c133700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58240fa950 con 0x7f5824070430 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f5829ecf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f5829ecf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50584/0 (socket says 172.21.15.87:50584) 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f5829ecf700 1 -- 172.21.15.87:0/3242016374 learned_addr learned my addr 172.21.15.87:0/3242016374 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f5829ecf700 1 -- 172.21.15.87:0/3242016374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581400b200 con 0x7f5824070430 2024-02-20T00:25:25.949 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f5829ecf700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5814006800 tx=0x7f5814006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581401b570 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f581401bb60 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.945+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58240faaf0 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581401a9f0 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58240faf80 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f581400f070 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f58127fc700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 0x7f580c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5814041460 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.946+0000 7f58296ce700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 0x7f580c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.947+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5824059100 con 0x7f5824070430 2024-02-20T00:25:25.950 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.947+0000 7f58296ce700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 0x7f580c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f58200096a0 tx=0x7f5820006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:25.955 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:25.951+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5814016070 con 0x7f5824070430 2024-02-20T00:25:26.104 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.102+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5824104af0 con 0x7f580c02fe70 2024-02-20T00:25:26.296 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.295+0000 7f58127fc700 1 -- 172.21.15.87:0/3242016374 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5824104af0 con 0x7f580c02fe70 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 msgr2=0x7f580c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 0x7f580c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f58200096a0 tx=0x7f5820006b40).stop 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 msgr2=0x7f58240fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5814006800 tx=0x7f5814006ee0).stop 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 shutdown_connections 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f580c02fe70 0x7f580c032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.296+0000 7f582c133700 1 --2- 172.21.15.87:0/3242016374 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5824070430 0x7f58240fc4d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.297+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 >> 172.21.15.87:0/3242016374 conn(0x7f58240f6530 msgr2=0x7f58240f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.297+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 shutdown_connections 2024-02-20T00:25:26.299 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:26.297+0000 7f582c133700 1 -- 172.21.15.87:0/3242016374 wait complete. 2024-02-20T00:25:26.815 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.312+0000 7f9be3fc7700 1 Processor -- start 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.312+0000 7f9be3fc7700 1 -- start start 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.312+0000 7f9be3fc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.312+0000 7f9be3fc7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0fc880 con 0x7f9bdc0703f0 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be1d63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be1d63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50594/0 (socket says 172.21.15.87:50594) 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be1d63700 1 -- 172.21.15.87:0/328664643 learned_addr learned my addr 172.21.15.87:0/328664643 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be1d63700 1 -- 172.21.15.87:0/328664643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc101380 con 0x7f9bdc0703f0 2024-02-20T00:25:27.362 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be1d63700 1 --2- 172.21.15.87:0/328664643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bcc00b5f0 tx=0x7f9bcc006ee0).ready entity=mon.0 client_cookie=1ae49e36430bfc7e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be0d61700 1 -- 172.21.15.87:0/328664643 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc00d600 con 0x7f9bdc0703f0 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be0d61700 1 -- 172.21.15.87:0/328664643 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9bcc00dbf0 con 0x7f9bdc0703f0 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.313+0000 7f9be0d61700 1 -- 172.21.15.87:0/328664643 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc011ce0 con 0x7f9bdc0703f0 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- 172.21.15.87:0/328664643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 msgr2=0x7f9bdc100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/328664643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bcc00b5f0 tx=0x7f9bcc006ee0).stop 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- 172.21.15.87:0/328664643 shutdown_connections 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/328664643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc100e90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- 172.21.15.87:0/328664643 >> 172.21.15.87:0/328664643 conn(0x7f9bdc0f6530 msgr2=0x7f9bdc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- 172.21.15.87:0/328664643 shutdown_connections 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- 172.21.15.87:0/328664643 wait complete. 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 Processor -- start 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- start start 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:27.363 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be3fc7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc10d290 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be1d63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.314+0000 7f9be1d63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50608/0 (socket says 172.21.15.87:50608) 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9be1d63700 1 -- 172.21.15.87:0/4106154856 learned_addr learned my addr 172.21.15.87:0/4106154856 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9be1d63700 1 -- 172.21.15.87:0/4106154856 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc00b200 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9be1d63700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9bcc011ed0 tx=0x7f9bcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc01b570 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc10d490 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9bcc01bb60 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc01aad0 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.315+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc10e0d0 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9bcc00f070 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9bca7fc700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 0x7f9bc40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9bcc041490 con 0x7f9bdc0703f0 2024-02-20T00:25:27.364 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc10d920 con 0x7f9bdc0703f0 2024-02-20T00:25:27.365 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9be1562700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 0x7f9bc40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:27.365 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.316+0000 7f9be1562700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 0x7f9bc40322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9bd80096a0 tx=0x7f9bd8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:27.365 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.320+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bcc016070 con 0x7f9bdc0703f0 2024-02-20T00:25:27.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.471+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bdc1020e0 con 0x7f9bc402fe10 2024-02-20T00:25:27.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.472+0000 7f9bca7fc700 1 -- 172.21.15.87:0/4106154856 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9bdc1020e0 con 0x7f9bc402fe10 2024-02-20T00:25:27.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLL/1XQiCIsRkr9P+51P6XzgH6FvwBZE4rPDOmxNa+uBHyNG1NTmgKSuMXSEXqgDN4amddvN6dP6EKSxx+lcMeOBSHY9Wk/eYK5f7HtPAU4zdt42+cMgB2pIHPMRoJd39OcZVnXBSaLKadUddpEfpEhMSna0oWVCwTXuospGtihc0SL4N3fnz5Bx3jhDAtECODftMbzJAqu7XJyge+vWCkNRKZCZos+05fgMQznXppqKyKl14HoLBTg50P9Eya6+rK/n+eIPG+HK0zIIviohAA4vnC9jderiJlfYb89dlHDIuMGhPAdagn6jhWzSc9O8i863dy+tXLhxIwXPXteCXH3hKVOLI70mmcfgrsIAbG5/J46n+AXXn3j26GrwEKeDXhDl2tgBaNYLQv6qh0FFU2LXVpMWiJyPmyMWM2R/sJOiB3q6laaTNijoE+x8+uVHZTItNBYzMMs6luEriftsowZrJOy8VSy4YLb6IKYsgGd7fSwiG1eTe2Qmz4kCLuKqs= ceph-6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 msgr2=0x7f9bc40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 0x7f9bc40322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9bd80096a0 tx=0x7f9bd8006b40).stop 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 msgr2=0x7f9bdc10cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9bcc011ed0 tx=0x7f9bcc006ee0).stop 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 shutdown_connections 2024-02-20T00:25:27.476 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f9bc402fe10 0x7f9bc40322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:27.477 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 --2- 172.21.15.87:0/4106154856 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9bdc0703f0 0x7f9bdc10cda0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:27.477 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 >> 172.21.15.87:0/4106154856 conn(0x7f9bdc0f6530 msgr2=0x7f9bdc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:27.477 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 shutdown_connections 2024-02-20T00:25:27.477 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:27.473+0000 7f9be3fc7700 1 -- 172.21.15.87:0/4106154856 wait complete. 2024-02-20T00:25:27.777 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:27 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:26.103402+0000 mgr.smithi087.iydsqo (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-20T00:25:27.777 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:27 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:26.293259+0000 mon.smithi087 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:27.777 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:27 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:26.294914+0000 mon.smithi087 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:28.031 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:25:28.031 INFO:teuthology.orchestra.run.smithi087.stderr:Adding key to root@localhost authorized_keys... 2024-02-20T00:25:28.032 INFO:teuthology.orchestra.run.smithi087.stderr:Adding host smithi087... 2024-02-20T00:25:28.032 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi087 172.21.15.87 2024-02-20T00:25:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.574+0000 7f5a28abb700 1 Processor -- start 2024-02-20T00:25:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.574+0000 7f5a28abb700 1 -- start start 2024-02-20T00:25:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.574+0000 7f5a28abb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.574+0000 7f5a28abb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240fef80 con 0x7f5a240703f0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52016/0 (socket says 172.21.15.87:52016) 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2259c700 1 -- 172.21.15.87:0/1766011416 learned_addr learned my addr 172.21.15.87:0/1766011416 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2259c700 1 -- 172.21.15.87:0/1766011416 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240ff0c0 con 0x7f5a240703f0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2259c700 1 --2- 172.21.15.87:0/1766011416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5a1800b5f0 tx=0x7f5a18006ee0).ready entity=mon.0 client_cookie=97f6bb3e2f2582fd server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2159a700 1 -- 172.21.15.87:0/1766011416 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1800d600 con 0x7f5a240703f0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2159a700 1 -- 172.21.15.87:0/1766011416 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5a1800dbf0 con 0x7f5a240703f0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.575+0000 7f5a2159a700 1 -- 172.21.15.87:0/1766011416 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a18011ce0 con 0x7f5a240703f0 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 -- 172.21.15.87:0/1766011416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 msgr2=0x7f5a240fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 --2- 172.21.15.87:0/1766011416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5a1800b5f0 tx=0x7f5a18006ee0).stop 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 -- 172.21.15.87:0/1766011416 shutdown_connections 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 --2- 172.21.15.87:0/1766011416 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a240fea90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 -- 172.21.15.87:0/1766011416 >> 172.21.15.87:0/1766011416 conn(0x7f5a240f6530 msgr2=0x7f5a240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 -- 172.21.15.87:0/1766011416 shutdown_connections 2024-02-20T00:25:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.576+0000 7f5a28abb700 1 -- 172.21.15.87:0/1766011416 wait complete. 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a28abb700 1 Processor -- start 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a28abb700 1 -- start start 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a28abb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a28abb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2410b010 con 0x7f5a240703f0 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a2259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a2259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52030/0 (socket says 172.21.15.87:52030) 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a2259c700 1 -- 172.21.15.87:0/448309222 learned_addr learned my addr 172.21.15.87:0/448309222 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a2259c700 1 -- 172.21.15.87:0/448309222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1800b200 con 0x7f5a240703f0 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a2259c700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5a18011ed0 tx=0x7f5a18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.577+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1801b570 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5a1801bb60 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1801aad0 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2410b210 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2410b6a0 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5a1800f070 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a12ffd700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 0x7f5a08032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.578+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5a18041300 con 0x7f5a240703f0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.579+0000 7f5a21d9b700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 0x7f5a08032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.579+0000 7f5a21d9b700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 0x7f5a08032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5a140096a0 tx=0x7f5a14006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.579+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24045ad0 con 0x7f5a240703f0 2024-02-20T00:25:28.585 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.583+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a18016070 con 0x7f5a240703f0 2024-02-20T00:25:28.755 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:28.753+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a24104c00 con 0x7f5a0802fe70 2024-02-20T00:25:29.352 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.350+0000 7f5a12ffd700 1 -- 172.21.15.87:0/448309222 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f5a24104c00 con 0x7f5a0802fe70 2024-02-20T00:25:29.352 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Added host 'smithi087' 2024-02-20T00:25:29.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 msgr2=0x7f5a08032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:29.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 0x7f5a08032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5a140096a0 tx=0x7f5a14006b40).stop 2024-02-20T00:25:29.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 msgr2=0x7f5a2410ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:29.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5a18011ed0 tx=0x7f5a18006ee0).stop 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 shutdown_connections 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f5a0802fe70 0x7f5a08032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 --2- 172.21.15.87:0/448309222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5a240703f0 0x7f5a2410ab20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 >> 172.21.15.87:0/448309222 conn(0x7f5a240f6530 msgr2=0x7f5a240f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 shutdown_connections 2024-02-20T00:25:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:29.352+0000 7f5a28abb700 1 -- 172.21.15.87:0/448309222 wait complete. 2024-02-20T00:25:29.380 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:27.472759+0000 mgr.smithi087.iydsqo (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:29.987 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying mon service with default placement... 2024-02-20T00:25:29.988 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-20T00:25:30.278 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:28.755257+0000 mgr.smithi087.iydsqo (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:30.279 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:29.350233+0000 mon.smithi087 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:30.279 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:29.350734+0000 mon.smithi087 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:30.519 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.517+0000 7fca33b69700 1 Processor -- start 2024-02-20T00:25:30.522 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.517+0000 7fca33b69700 1 -- start start 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.517+0000 7fca33b69700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.517+0000 7fca33b69700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0fef40 con 0x7fca2c070430 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca31905700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca31905700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52036/0 (socket says 172.21.15.87:52036) 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca31905700 1 -- 172.21.15.87:0/4065176882 learned_addr learned my addr 172.21.15.87:0/4065176882 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca31905700 1 -- 172.21.15.87:0/4065176882 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0ff080 con 0x7fca2c070430 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca31905700 1 --2- 172.21.15.87:0/4065176882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fca1c00b5f0 tx=0x7fca1c006ee0).ready entity=mon.0 client_cookie=6e4fd3e2bc81e05e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca30903700 1 -- 172.21.15.87:0/4065176882 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c00d600 con 0x7fca2c070430 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca30903700 1 -- 172.21.15.87:0/4065176882 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fca1c00dbf0 con 0x7fca2c070430 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.518+0000 7fca30903700 1 -- 172.21.15.87:0/4065176882 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c011cb0 con 0x7fca2c070430 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.519+0000 7fca33b69700 1 -- 172.21.15.87:0/4065176882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 msgr2=0x7fca2c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.519+0000 7fca33b69700 1 --2- 172.21.15.87:0/4065176882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fca1c00b5f0 tx=0x7fca1c006ee0).stop 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.519+0000 7fca33b69700 1 -- 172.21.15.87:0/4065176882 shutdown_connections 2024-02-20T00:25:30.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.519+0000 7fca33b69700 1 --2- 172.21.15.87:0/4065176882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.519+0000 7fca33b69700 1 -- 172.21.15.87:0/4065176882 >> 172.21.15.87:0/4065176882 conn(0x7fca2c0f6530 msgr2=0x7fca2c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 -- 172.21.15.87:0/4065176882 shutdown_connections 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 -- 172.21.15.87:0/4065176882 wait complete. 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 Processor -- start 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 -- start start 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca33b69700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0fa950 con 0x7fca2c070430 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca31905700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca31905700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52038/0 (socket says 172.21.15.87:52038) 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.520+0000 7fca31905700 1 -- 172.21.15.87:0/3201210986 learned_addr learned my addr 172.21.15.87:0/3201210986 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:30.524 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca31905700 1 -- 172.21.15.87:0/3201210986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c00b200 con 0x7fca2c070430 2024-02-20T00:25:30.525 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca31905700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fca1c006800 tx=0x7fca1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:30.525 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c01b570 con 0x7fca2c070430 2024-02-20T00:25:30.525 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fca1c01bb60 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c0faaf0 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c01a9f0 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.521+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c0faf80 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.522+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fca1c00f070 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.522+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c045ad0 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.522+0000 7fca1a7fc700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 0x7fca14032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.522+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fca1c041380 con 0x7fca2c070430 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.522+0000 7fca31104700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 0x7fca14032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:30.526 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.523+0000 7fca31104700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 0x7fca14032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:30.528 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.526+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca1c019cd0 con 0x7fca2c070430 2024-02-20T00:25:30.703 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.701+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fca2c0fb1c0 con 0x7fca1402fe70 2024-02-20T00:25:30.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.705+0000 7fca1a7fc700 1 -- 172.21.15.87:0/3201210986 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fca2c0fb1c0 con 0x7fca1402fe70 2024-02-20T00:25:30.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-20T00:25:30.710 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 msgr2=0x7fca14032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:30.710 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 0x7fca14032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).stop 2024-02-20T00:25:30.710 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 msgr2=0x7fca2c0fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:30.710 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fca1c006800 tx=0x7fca1c006ee0).stop 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 shutdown_connections 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7fca1402fe70 0x7fca14032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 --2- 172.21.15.87:0/3201210986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fca2c070430 0x7fca2c0fc4d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 >> 172.21.15.87:0/3201210986 conn(0x7fca2c0f6530 msgr2=0x7fca2c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 shutdown_connections 2024-02-20T00:25:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:30.707+0000 7fca33b69700 1 -- 172.21.15.87:0/3201210986 wait complete. 2024-02-20T00:25:31.346 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying mgr service with default placement... 2024-02-20T00:25:31.346 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-20T00:25:31.349 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:29.350504+0000 mgr.smithi087.iydsqo (mgr.14114) 9 : cephadm [INF] Added host smithi087 2024-02-20T00:25:31.349 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:30.706212+0000 mon.smithi087 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:31.889 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b9090700 1 Processor -- start 2024-02-20T00:25:31.889 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b9090700 1 -- start start 2024-02-20T00:25:31.889 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b9090700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b9090700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40fc8c0 con 0x7f07b40703b0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b2d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b2d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52042/0 (socket says 172.21.15.87:52042) 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.885+0000 7f07b2d9d700 1 -- 172.21.15.87:0/3021244172 learned_addr learned my addr 172.21.15.87:0/3021244172 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.886+0000 7f07b2d9d700 1 -- 172.21.15.87:0/3021244172 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b41013c0 con 0x7f07b40703b0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.886+0000 7f07b2d9d700 1 --2- 172.21.15.87:0/3021244172 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f07a800b5f0 tx=0x7f07a8006ee0).ready entity=mon.0 client_cookie=3582acdd563c0ab7 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.886+0000 7f07b1d9b700 1 -- 172.21.15.87:0/3021244172 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a800d600 con 0x7f07b40703b0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.886+0000 7f07b1d9b700 1 -- 172.21.15.87:0/3021244172 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f07a800dbf0 con 0x7f07b40703b0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.886+0000 7f07b1d9b700 1 -- 172.21.15.87:0/3021244172 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a8011cb0 con 0x7f07b40703b0 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- 172.21.15.87:0/3021244172 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 msgr2=0x7f07b4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 --2- 172.21.15.87:0/3021244172 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f07a800b5f0 tx=0x7f07a8006ee0).stop 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- 172.21.15.87:0/3021244172 shutdown_connections 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 --2- 172.21.15.87:0/3021244172 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b4100ed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:31.890 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- 172.21.15.87:0/3021244172 >> 172.21.15.87:0/3021244172 conn(0x7f07b40f6530 msgr2=0x7f07b40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:31.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- 172.21.15.87:0/3021244172 shutdown_connections 2024-02-20T00:25:31.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- 172.21.15.87:0/3021244172 wait complete. 2024-02-20T00:25:31.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 Processor -- start 2024-02-20T00:25:31.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.887+0000 7f07b9090700 1 -- start start 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b9090700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b9090700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b410d270 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b2d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b2d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52058/0 (socket says 172.21.15.87:52058) 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b2d9d700 1 -- 172.21.15.87:0/935125781 learned_addr learned my addr 172.21.15.87:0/935125781 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b2d9d700 1 -- 172.21.15.87:0/935125781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a800b200 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b2d9d700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f07a8006800 tx=0x7f07a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a801b570 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f07a801bb60 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b410d470 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a801a9f0 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.888+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b410e080 con 0x7f07b40703b0 2024-02-20T00:25:31.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.889+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f07a800f070 con 0x7f07b40703b0 2024-02-20T00:25:31.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.889+0000 7f079b7fe700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 0x7f07a00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:31.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.889+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f07a80414a0 con 0x7f07b40703b0 2024-02-20T00:25:31.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.889+0000 7f07b259c700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 0x7f07a00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:31.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.890+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4059100 con 0x7f07b40703b0 2024-02-20T00:25:31.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.890+0000 7f07b259c700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 0x7f07a00322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f07a40096a0 tx=0x7f07a4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:31.895 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:31.894+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07a8016070 con 0x7f07b40703b0 2024-02-20T00:25:32.072 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.070+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f07b4102140 con 0x7f07a002fe10 2024-02-20T00:25:32.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.074+0000 7f079b7fe700 1 -- 172.21.15.87:0/935125781 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f07b4102140 con 0x7f07a002fe10 2024-02-20T00:25:32.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-20T00:25:32.081 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 msgr2=0x7f07a00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 0x7f07a00322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f07a40096a0 tx=0x7f07a4006b40).stop 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 msgr2=0x7f07b410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f07a8006800 tx=0x7f07a8006ee0).stop 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 shutdown_connections 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f07a002fe10 0x7f07a00322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 --2- 172.21.15.87:0/935125781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07b40703b0 0x7f07b410cd80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 >> 172.21.15.87:0/935125781 conn(0x7f07b40f6530 msgr2=0x7f07b40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 shutdown_connections 2024-02-20T00:25:32.082 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:32.076+0000 7f07b9090700 1 -- 172.21.15.87:0/935125781 wait complete. 2024-02-20T00:25:32.448 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:30.703350+0000 mgr.smithi087.iydsqo (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:32.449 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:30.704074+0000 mgr.smithi087.iydsqo (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-20T00:25:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying crash service with default placement... 2024-02-20T00:25:32.711 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.216+0000 7febea153700 1 Processor -- start 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.216+0000 7febea153700 1 -- start start 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febea153700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febea153700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40fdeb0 con 0x7febe40fd5a0 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52064/0 (socket says 172.21.15.87:52064) 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe37fe700 1 -- 172.21.15.87:0/3920468950 learned_addr learned my addr 172.21.15.87:0/3920468950 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe37fe700 1 -- 172.21.15.87:0/3920468950 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40fdff0 con 0x7febe40fd5a0 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe37fe700 1 --2- 172.21.15.87:0/3920468950 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7febd800b5f0 tx=0x7febd8006ee0).ready entity=mon.0 client_cookie=44612d9bfe6f4fd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:33.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe27fc700 1 -- 172.21.15.87:0/3920468950 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd800d600 con 0x7febe40fd5a0 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe27fc700 1 -- 172.21.15.87:0/3920468950 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7febd800dbf0 con 0x7febe40fd5a0 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.217+0000 7febe27fc700 1 -- 172.21.15.87:0/3920468950 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd8011cb0 con 0x7febe40fd5a0 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 -- 172.21.15.87:0/3920468950 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 msgr2=0x7febe40fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 --2- 172.21.15.87:0/3920468950 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7febd800b5f0 tx=0x7febd8006ee0).stop 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 -- 172.21.15.87:0/3920468950 shutdown_connections 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 --2- 172.21.15.87:0/3920468950 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fd9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 -- 172.21.15.87:0/3920468950 >> 172.21.15.87:0/3920468950 conn(0x7febe40f5d20 msgr2=0x7febe40f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 -- 172.21.15.87:0/3920468950 shutdown_connections 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 -- 172.21.15.87:0/3920468950 wait complete. 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.218+0000 7febea153700 1 Processor -- start 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febea153700 1 -- start start 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febea153700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febea153700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40fb2f0 con 0x7febe40fd5a0 2024-02-20T00:25:33.223 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febe37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febe37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52072/0 (socket says 172.21.15.87:52072) 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febe37fe700 1 -- 172.21.15.87:0/3884186045 learned_addr learned my addr 172.21.15.87:0/3884186045 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febe37fe700 1 -- 172.21.15.87:0/3884186045 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd800b200 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febe37fe700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7febd8002f40 tx=0x7febd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd801b410 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7febd801ba00 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd801a980 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.219+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40fb490 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.220+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe40fb920 con 0x7febe40fd5a0 2024-02-20T00:25:33.224 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.220+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7febd800f070 con 0x7febe40fd5a0 2024-02-20T00:25:33.225 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.220+0000 7febcbfff700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 0x7febd0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:33.225 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.220+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7febd8041440 con 0x7febe40fd5a0 2024-02-20T00:25:33.225 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.221+0000 7febe2ffd700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 0x7febd0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:33.225 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.221+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe4059100 con 0x7febe40fd5a0 2024-02-20T00:25:33.225 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.221+0000 7febe2ffd700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 0x7febd0032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7febd40096a0 tx=0x7febd4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:33.226 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.225+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febd8016020 con 0x7febe40fd5a0 2024-02-20T00:25:33.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.401+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7febe4104710 con 0x7febd002fe70 2024-02-20T00:25:33.406 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.404+0000 7febcbfff700 1 -- 172.21.15.87:0/3884186045 <== mgr.14114 v2:172.21.15.87:6800/67270349 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7febe4104710 con 0x7febd002fe70 2024-02-20T00:25:33.406 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 msgr2=0x7febd0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 0x7febd0032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7febd40096a0 tx=0x7febd4006b40).stop 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 msgr2=0x7febe40fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7febd8002f40 tx=0x7febd8006ee0).stop 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 shutdown_connections 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7febd002fe70 0x7febd0032330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 --2- 172.21.15.87:0/3884186045 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7febe40fd5a0 0x7febe40fce70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:33.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 >> 172.21.15.87:0/3884186045 conn(0x7febe40f5d20 msgr2=0x7febe40ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:33.410 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.406+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 shutdown_connections 2024-02-20T00:25:33.410 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:33.407+0000 7febea153700 1 -- 172.21.15.87:0/3884186045 wait complete. 2024-02-20T00:25:33.432 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:32.072298+0000 mgr.smithi087.iydsqo (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:33.432 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:32.072805+0000 mgr.smithi087.iydsqo (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-20T00:25:33.432 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:32.074634+0000 mon.smithi087 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling mgr prometheus module... 2024-02-20T00:25:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-20T00:25:34.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:33.402484+0000 mgr.smithi087.iydsqo (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:34.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:33.403098+0000 mgr.smithi087.iydsqo (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-20T00:25:34.536 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:33.405407+0000 mon.smithi087 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.87:0/2394540460' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:34.673 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.666+0000 7f87c9eed700 1 Processor -- start 2024-02-20T00:25:34.673 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.666+0000 7f87c9eed700 1 -- start start 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c9eed700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c9eed700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40fef80 con 0x7f87c40703f0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52076/0 (socket says 172.21.15.87:52076) 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c37fe700 1 -- 172.21.15.87:0/984292135 learned_addr learned my addr 172.21.15.87:0/984292135 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c37fe700 1 -- 172.21.15.87:0/984292135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c40ff0c0 con 0x7f87c40703f0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c37fe700 1 --2- 172.21.15.87:0/984292135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f87b800b5f0 tx=0x7f87b8006ee0).ready entity=mon.0 client_cookie=1a2ce7ee0baa314c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c27fc700 1 -- 172.21.15.87:0/984292135 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b800d600 con 0x7f87c40703f0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c27fc700 1 -- 172.21.15.87:0/984292135 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f87b800dbf0 con 0x7f87c40703f0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.667+0000 7f87c27fc700 1 -- 172.21.15.87:0/984292135 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b8011ce0 con 0x7f87c40703f0 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 -- 172.21.15.87:0/984292135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 msgr2=0x7f87c40fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 --2- 172.21.15.87:0/984292135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f87b800b5f0 tx=0x7f87b8006ee0).stop 2024-02-20T00:25:34.674 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 -- 172.21.15.87:0/984292135 shutdown_connections 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 --2- 172.21.15.87:0/984292135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fea90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 -- 172.21.15.87:0/984292135 >> 172.21.15.87:0/984292135 conn(0x7f87c40f6530 msgr2=0x7f87c40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 -- 172.21.15.87:0/984292135 shutdown_connections 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.668+0000 7f87c9eed700 1 -- 172.21.15.87:0/984292135 wait complete. 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c9eed700 1 Processor -- start 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c9eed700 1 -- start start 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c9eed700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c9eed700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40fa990 con 0x7f87c40703f0 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52088/0 (socket says 172.21.15.87:52088) 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.669+0000 7f87c37fe700 1 -- 172.21.15.87:0/2672884678 learned_addr learned my addr 172.21.15.87:0/2672884678 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:34.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87c37fe700 1 -- 172.21.15.87:0/2672884678 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b800b200 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87c37fe700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f87b80115c0 tx=0x7f87b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b801b410 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f87b801ba00 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c40fab30 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b801aaa0 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.670+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c40fafc0 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f87b800f070 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87abfff700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 0x7f87b0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f87b8041010 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c4045ad0 con 0x7f87c40703f0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87c2ffd700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 0x7f87b0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:34.676 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.671+0000 7f87c2ffd700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 0x7f87b0032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f87b40096a0 tx=0x7f87b4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:34.677 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.675+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87b8010be0 con 0x7f87c40703f0 2024-02-20T00:25:34.865 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:34.863+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f87c40ffe00 con 0x7f87c40703f0 2024-02-20T00:25:35.409 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.407+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f87b8024930 con 0x7f87c40703f0 2024-02-20T00:25:35.410 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.408+0000 7f87abfff700 1 -- 172.21.15.87:0/2672884678 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f87b80224f0 con 0x7f87c40703f0 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 msgr2=0x7f87b0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 0x7f87b0032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f87b40096a0 tx=0x7f87b4006b40).stop 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 msgr2=0x7f87c40fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f87b80115c0 tx=0x7f87b8006ee0).stop 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 shutdown_connections 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f87b002fe70 0x7f87b0032330 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 --2- 172.21.15.87:0/2672884678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f87c40703f0 0x7f87c40fc510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.411+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 >> 172.21.15.87:0/2672884678 conn(0x7f87c40f6530 msgr2=0x7f87c40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:35.414 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.412+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 shutdown_connections 2024-02-20T00:25:35.415 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:35.412+0000 7f87c9eed700 1 -- 172.21.15.87:0/2672884678 wait complete. 2024-02-20T00:25:35.771 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:35 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:34.864558+0000 mon.smithi087 (mon.0) 59 : audit [INF] from='client.? 172.21.15.87:0/2672884678' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-20T00:25:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying prometheus service with default placement... 2024-02-20T00:25:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-20T00:25:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.649+0000 7f50d13e5700 1 Processor -- start 2024-02-20T00:25:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.649+0000 7f50d13e5700 1 -- start start 2024-02-20T00:25:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.649+0000 7f50d13e5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.650+0000 7f50d13e5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc068ed0 con 0x7f50cc0685e0 2024-02-20T00:25:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.651+0000 7f50caffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.651+0000 7f50caffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52104/0 (socket says 172.21.15.87:52104) 2024-02-20T00:25:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.651+0000 7f50caffd700 1 -- 172.21.15.87:0/305726289 learned_addr learned my addr 172.21.15.87:0/305726289 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:36.675 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.673+0000 7f50caffd700 1 -- 172.21.15.87:0/305726289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc069010 con 0x7f50cc0685e0 2024-02-20T00:25:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.684+0000 7f50caffd700 1 --2- 172.21.15.87:0/305726289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f50bc00c700 tx=0x7f50bc006ee0).ready entity=mon.0 client_cookie=7a144443a8cd122 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50c9ffb700 1 -- 172.21.15.87:0/305726289 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc010780 con 0x7f50cc0685e0 2024-02-20T00:25:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50c9ffb700 1 -- 172.21.15.87:0/305726289 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f50bc010d70 con 0x7f50cc0685e0 2024-02-20T00:25:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50c9ffb700 1 -- 172.21.15.87:0/305726289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc00ee10 con 0x7f50cc0685e0 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50d13e5700 1 -- 172.21.15.87:0/305726289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 msgr2=0x7f50cc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50d13e5700 1 --2- 172.21.15.87:0/305726289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f50bc00c700 tx=0x7f50bc006ee0).stop 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50d13e5700 1 -- 172.21.15.87:0/305726289 shutdown_connections 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50d13e5700 1 --2- 172.21.15.87:0/305726289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc0689e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.701+0000 7f50d13e5700 1 -- 172.21.15.87:0/305726289 >> 172.21.15.87:0/305726289 conn(0x7f50cc063ba0 msgr2=0x7f50cc065fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.702+0000 7f50d13e5700 1 -- 172.21.15.87:0/305726289 shutdown_connections 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.702+0000 7f50d13e5700 1 -- 172.21.15.87:0/305726289 wait complete. 2024-02-20T00:25:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.702+0000 7f50d13e5700 1 Processor -- start 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.702+0000 7f50d13e5700 1 -- start start 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50d13e5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50d13e5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc105ea0 con 0x7f50cc0685e0 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50caffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50caffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52112/0 (socket says 172.21.15.87:52112) 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50caffd700 1 -- 172.21.15.87:0/2894231407 learned_addr learned my addr 172.21.15.87:0/2894231407 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50caffd700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc00c310 con 0x7f50cc0685e0 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50caffd700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f50bc000c00 tx=0x7f50bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:36.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc019630 con 0x7f50cc0685e0 2024-02-20T00:25:36.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f50bc01d070 con 0x7f50cc0685e0 2024-02-20T00:25:36.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50bc018b30 con 0x7f50cc0685e0 2024-02-20T00:25:36.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc106040 con 0x7f50cc0685e0 2024-02-20T00:25:36.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.703+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc1064d0 con 0x7f50cc0685e0 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.704+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f50bc029070 con 0x7f50cc0685e0 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.705+0000 7f50b37fe700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.705+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f50bc041340 con 0x7f50cc0685e0 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.705+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc045ad0 con 0x7f50cc0685e0 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.705+0000 7f50ca7fc700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/67270349 2024-02-20T00:25:36.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.705+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:25:36.714 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.712+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50bc014080 con 0x7f50cc0685e0 2024-02-20T00:25:36.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:35.409506+0000 mon.smithi087 (mon.0) 60 : audit [INF] from='client.? 172.21.15.87:0/2672884678' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-20T00:25:36.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:35.409548+0000 mon.smithi087 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi087.iydsqo(active, since 14s) 2024-02-20T00:25:36.901 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.899+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f50cc10d300 con 0x7f50b802ff10 2024-02-20T00:25:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.906+0000 7f50ca7fc700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/67270349 2024-02-20T00:25:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:36.906+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:25:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:37.307+0000 7f50ca7fc700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/67270349 2024-02-20T00:25:37.309 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:37.307+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:25:38.110 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:38.108+0000 7f50ca7fc700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/67270349 2024-02-20T00:25:38.110 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:38.108+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T00:25:39.712 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:39.708+0000 7f50ca7fc700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/67270349 2024-02-20T00:25:39.712 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:39.708+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-20T00:25:40.449 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:40.446+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f50bc00e040 con 0x7f50cc0685e0 2024-02-20T00:25:40.449 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:40.446+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 msgr2=0x7f50b80323d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:25:40.449 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:40.446+0000 7f50b37fe700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:40.395857+0000 mon.smithi087 (mon.0) 62 : cluster [INF] Active manager daemon smithi087.iydsqo restarted 2024-02-20T00:25:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:40.396307+0000 mon.smithi087 (mon.0) 63 : cluster [INF] Activating manager daemon smithi087.iydsqo 2024-02-20T00:25:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:40.398546+0000 mon.smithi087 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-20T00:25:41.457 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.453+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f50bc0183b0 con 0x7f50cc0685e0 2024-02-20T00:25:41.457 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.453+0000 7f50b37fe700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 0x7f50b8036120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:41.458 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.453+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 --> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f50cc10d300 con 0x7f50b8033d30 2024-02-20T00:25:41.458 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.453+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 0x7f50b8036120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:41.458 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.454+0000 7f50ca7fc700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 0x7f50b8036120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f50c0006fd0 tx=0x7f50c0006db0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:41.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.458+0000 7f50b37fe700 1 -- 172.21.15.87:0/2894231407 <== mgr.14142 v2:172.21.15.87:6800/3977108171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f50cc10d300 con 0x7f50b8033d30 2024-02-20T00:25:41.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-20T00:25:41.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 msgr2=0x7f50b8036120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:41.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 0x7f50b8036120 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f50c0006fd0 tx=0x7f50c0006db0).stop 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 msgr2=0x7f50cc107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f50bc000c00 tx=0x7f50bc006ee0).stop 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 shutdown_connections 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f50b8033d30 0x7f50b8036120 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50cc0685e0 0x7f50cc107a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 --2- 172.21.15.87:0/2894231407 >> [v2:172.21.15.87:6800/67270349,v1:172.21.15.87:6801/67270349] conn(0x7f50b802ff10 0x7f50b80323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.460+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 >> 172.21.15.87:0/2894231407 conn(0x7f50cc063ba0 msgr2=0x7f50cc066540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.461+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 shutdown_connections 2024-02-20T00:25:41.465 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:41.461+0000 7f50d13e5700 1 -- 172.21.15.87:0/2894231407 wait complete. 2024-02-20T00:25:41.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:40.449077+0000 mon.smithi087 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi087.iydsqo(active, starting, since 0.0528591s) 2024-02-20T00:25:41.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.451523+0000 mon.smithi087 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata", "id": "smithi087"}]: dispatch 2024-02-20T00:25:41.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.452172+0000 mon.smithi087 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mgr metadata", "who": "smithi087.iydsqo", "id": "smithi087.iydsqo"}]: dispatch 2024-02-20T00:25:41.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.452818+0000 mon.smithi087 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:25:41.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.452961+0000 mon.smithi087 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.453098+0000 mon.smithi087 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:40.455017+0000 mon.smithi087 (mon.0) 71 : cluster [INF] Manager daemon smithi087.iydsqo is now available 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.551307+0000 mon.smithi087 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.552831+0000 mon.smithi087 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.572029+0000 mon.smithi087 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:25:41.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:40.575750+0000 mon.smithi087 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:25:41.983 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying grafana service with default placement... 2024-02-20T00:25:41.983 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-20T00:25:42.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:41.456937+0000 mon.smithi087 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi087.iydsqo(active, since 1.06072s) 2024-02-20T00:25:42.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:41.457840+0000 mgr.smithi087.iydsqo (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-20T00:25:42.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:41.459934+0000 mon.smithi087 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:42.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.587+0000 7fa9d0e2f700 1 Processor -- start 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.587+0000 7fa9d0e2f700 1 -- start start 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.587+0000 7fa9d0e2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9d0e2f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc0fc8c0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43006/0 (socket says 172.21.15.87:43006) 2024-02-20T00:25:42.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9ca59c700 1 -- 172.21.15.87:0/1589295006 learned_addr learned my addr 172.21.15.87:0/1589295006 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9ca59c700 1 -- 172.21.15.87:0/1589295006 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc1013c0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9ca59c700 1 --2- 172.21.15.87:0/1589295006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa9c000b5f0 tx=0x7fa9c0006ee0).ready entity=mon.0 client_cookie=8f1e157f43b4d3b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9c959a700 1 -- 172.21.15.87:0/1589295006 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c000d600 con 0x7fa9cc0703b0 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9c959a700 1 -- 172.21.15.87:0/1589295006 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa9c000dbf0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.588+0000 7fa9c959a700 1 -- 172.21.15.87:0/1589295006 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c0011cb0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/1589295006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 msgr2=0x7fa9cc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/1589295006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa9c000b5f0 tx=0x7fa9c0006ee0).stop 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/1589295006 shutdown_connections 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/1589295006 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc100ed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:42.596 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/1589295006 >> 172.21.15.87:0/1589295006 conn(0x7fa9cc0f6530 msgr2=0x7fa9cc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/1589295006 shutdown_connections 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.589+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/1589295006 wait complete. 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9d0e2f700 1 Processor -- start 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9d0e2f700 1 -- start start 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9d0e2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9d0e2f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc10d2c0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43022/0 (socket says 172.21.15.87:43022) 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9ca59c700 1 -- 172.21.15.87:0/2777900355 learned_addr learned my addr 172.21.15.87:0/2777900355 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9ca59c700 1 -- 172.21.15.87:0/2777900355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c000b200 con 0x7fa9cc0703b0 2024-02-20T00:25:42.597 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9ca59c700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa9c000df90 tx=0x7fa9c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c001b410 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa9c001ba00 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc10d4c0 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.590+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c001aa30 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.591+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc10e100 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.591+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa9c000f070 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.591+0000 7fa9baffd700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 0x7fa9b00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.591+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa9c0046630 con 0x7fa9cc0703b0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.591+0000 7fa9c9d9b700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 0x7fa9b00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.592+0000 7fa9c9d9b700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 0x7fa9b00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa9bc0096a0 tx=0x7fa9bc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.592+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc059100 con 0x7fa9cc0703b0 2024-02-20T00:25:42.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.596+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9c0016020 con 0x7fa9cc0703b0 2024-02-20T00:25:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.769+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 --> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9cc102180 con 0x7fa9b002ff20 2024-02-20T00:25:42.778 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.775+0000 7fa9baffd700 1 -- 172.21.15.87:0/2777900355 <== mgr.14142 v2:172.21.15.87:6800/3977108171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa9cc102180 con 0x7fa9b002ff20 2024-02-20T00:25:42.778 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 msgr2=0x7fa9b00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 0x7fa9b00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa9bc0096a0 tx=0x7fa9bc006b40).stop 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 msgr2=0x7fa9cc10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa9c000df90 tx=0x7fa9c0006ee0).stop 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 shutdown_connections 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fa9b002ff20 0x7fa9b00323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 --2- 172.21.15.87:0/2777900355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9cc0703b0 0x7fa9cc10cdd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:42.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.777+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 >> 172.21.15.87:0/2777900355 conn(0x7fa9cc0f6530 msgr2=0x7fa9cc0f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:42.782 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.778+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 shutdown_connections 2024-02-20T00:25:42.782 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:42.778+0000 7fa9d0e2f700 1 -- 172.21.15.87:0/2777900355 wait complete. 2024-02-20T00:25:43.406 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying node-exporter service with default placement... 2024-02-20T00:25:43.406 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-20T00:25:43.659 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:43 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:42.461234+0000 mon.smithi087 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi087.iydsqo(active, since 2s) 2024-02-20T00:25:43.659 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:43 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:42.771726+0000 mgr.smithi087.iydsqo (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:43.659 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:43 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:42.772275+0000 mgr.smithi087.iydsqo (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-20T00:25:43.660 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:43 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:42.777440+0000 mon.smithi087 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:43.965 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.943+0000 7f5ed2460700 1 Processor -- start 2024-02-20T00:25:43.965 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.943+0000 7f5ed2460700 1 -- start start 2024-02-20T00:25:43.965 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.943+0000 7f5ed2460700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.943+0000 7f5ed2460700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0fb710 con 0x7f5ecc0fae00 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43038/0 (socket says 172.21.15.87:43038) 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecbfff700 1 -- 172.21.15.87:0/4046117825 learned_addr learned my addr 172.21.15.87:0/4046117825 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecbfff700 1 -- 172.21.15.87:0/4046117825 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0fb850 con 0x7f5ecc0fae00 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecbfff700 1 --2- 172.21.15.87:0/4046117825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5ec000b5f0 tx=0x7f5ec0006ee0).ready entity=mon.0 client_cookie=b89a7bac6a1c97 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecaffd700 1 -- 172.21.15.87:0/4046117825 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec000d600 con 0x7f5ecc0fae00 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecaffd700 1 -- 172.21.15.87:0/4046117825 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5ec000dbf0 con 0x7f5ecc0fae00 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.944+0000 7f5ecaffd700 1 -- 172.21.15.87:0/4046117825 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec0011cb0 con 0x7f5ecc0fae00 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- 172.21.15.87:0/4046117825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 msgr2=0x7f5ecc0fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 --2- 172.21.15.87:0/4046117825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5ec000b5f0 tx=0x7f5ec0006ee0).stop 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- 172.21.15.87:0/4046117825 shutdown_connections 2024-02-20T00:25:43.966 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 --2- 172.21.15.87:0/4046117825 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc0fb220 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- 172.21.15.87:0/4046117825 >> 172.21.15.87:0/4046117825 conn(0x7f5ecc0f65b0 msgr2=0x7f5ecc0f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- 172.21.15.87:0/4046117825 shutdown_connections 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- 172.21.15.87:0/4046117825 wait complete. 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 Processor -- start 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.945+0000 7f5ed2460700 1 -- start start 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ed2460700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ed2460700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc06fc30 con 0x7f5ecc0fae00 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ecbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43054/0 (socket says 172.21.15.87:43054) 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ecbfff700 1 -- 172.21.15.87:0/1141121497 learned_addr learned my addr 172.21.15.87:0/1141121497 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ecbfff700 1 -- 172.21.15.87:0/1141121497 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec000b200 con 0x7f5ecc0fae00 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ecbfff700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5ec0006800 tx=0x7f5ec0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec001b570 con 0x7f5ecc0fae00 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5ec001bb60 con 0x7f5ecc0fae00 2024-02-20T00:25:43.967 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc06fe30 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec001a9f0 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.946+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc10df60 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f5ec000f070 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ec8ff9700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 0x7f5eb80323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5ec0041640 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc045ad0 con 0x7f5ecc0fae00 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ecb7fe700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 0x7f5eb80323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.947+0000 7f5ecb7fe700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 0x7f5eb80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5ebc0096a0 tx=0x7f5ebc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:43.968 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:43.951+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ec0016070 con 0x7f5ecc0fae00 2024-02-20T00:25:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.131+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 --> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ecc0fcc90 con 0x7f5eb802ff20 2024-02-20T00:25:44.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.135+0000 7f5ec8ff9700 1 -- 172.21.15.87:0/1141121497 <== mgr.14142 v2:172.21.15.87:6800/3977108171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f5ecc0fcc90 con 0x7f5eb802ff20 2024-02-20T00:25:44.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 msgr2=0x7f5eb80323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 0x7f5eb80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5ebc0096a0 tx=0x7f5ebc006b40).stop 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 msgr2=0x7f5ecc10d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5ec0006800 tx=0x7f5ec0006ee0).stop 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 shutdown_connections 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f5eb802ff20 0x7f5eb80323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 --2- 172.21.15.87:0/1141121497 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5ecc0fae00 0x7f5ecc10d8b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.137+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 >> 172.21.15.87:0/1141121497 conn(0x7f5ecc0f65b0 msgr2=0x7f5ecc0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:44.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.138+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 shutdown_connections 2024-02-20T00:25:44.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:44.138+0000 7f5ed2460700 1 -- 172.21.15.87:0/1141121497 wait complete. 2024-02-20T00:25:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying alertmanager service with default placement... 2024-02-20T00:25:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-20T00:25:45.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.321+0000 7fbb1d946700 1 Processor -- start 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.321+0000 7fbb1d946700 1 -- start start 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.321+0000 7fbb1d946700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.321+0000 7fbb1d946700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb180fc8c0 con 0x7fbb180703b0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43068/0 (socket says 172.21.15.87:43068) 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb16ffd700 1 -- 172.21.15.87:0/2822976107 learned_addr learned my addr 172.21.15.87:0/2822976107 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb16ffd700 1 -- 172.21.15.87:0/2822976107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb181013c0 con 0x7fbb180703b0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb16ffd700 1 --2- 172.21.15.87:0/2822976107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbb0000b5f0 tx=0x7fbb00006ee0).ready entity=mon.0 client_cookie=4abab781a5e1a32f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb15ffb700 1 -- 172.21.15.87:0/2822976107 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb0000d600 con 0x7fbb180703b0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb15ffb700 1 -- 172.21.15.87:0/2822976107 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb0000dbf0 con 0x7fbb180703b0 2024-02-20T00:25:45.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.322+0000 7fbb15ffb700 1 -- 172.21.15.87:0/2822976107 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb00011cb0 con 0x7fbb180703b0 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 -- 172.21.15.87:0/2822976107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 msgr2=0x7fbb18100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 --2- 172.21.15.87:0/2822976107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbb0000b5f0 tx=0x7fbb00006ee0).stop 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 -- 172.21.15.87:0/2822976107 shutdown_connections 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 --2- 172.21.15.87:0/2822976107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb18100ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 -- 172.21.15.87:0/2822976107 >> 172.21.15.87:0/2822976107 conn(0x7fbb180f6530 msgr2=0x7fbb180f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 -- 172.21.15.87:0/2822976107 shutdown_connections 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.323+0000 7fbb1d946700 1 -- 172.21.15.87:0/2822976107 wait complete. 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 Processor -- start 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 -- start start 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1810d270 con 0x7fbb180703b0 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43080/0 (socket says 172.21.15.87:43080) 2024-02-20T00:25:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb16ffd700 1 -- 172.21.15.87:0/3413164007 learned_addr learned my addr 172.21.15.87:0/3413164007 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb16ffd700 1 -- 172.21.15.87:0/3413164007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0000b200 con 0x7fbb180703b0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb16ffd700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbb00006800 tx=0x7fbb00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb0001b570 con 0x7fbb180703b0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1810d470 con 0x7fbb180703b0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb0001bb60 con 0x7fbb180703b0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb0001a9f0 con 0x7fbb180703b0 2024-02-20T00:25:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.324+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1810e080 con 0x7fbb180703b0 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.325+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbb0000f070 con 0x7fbb180703b0 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.325+0000 7fbb0f7fe700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 0x7fbaf80323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.326+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbb000415e0 con 0x7fbb180703b0 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.326+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb18059100 con 0x7fbb180703b0 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.326+0000 7fbb167fc700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 0x7fbaf80323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.326+0000 7fbb167fc700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 0x7fbaf80323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb080096a0 tx=0x7fbb08006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.330+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb00016070 con 0x7fbb180703b0 2024-02-20T00:25:45.509 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.507+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 --> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb18102150 con 0x7fbaf802ff20 2024-02-20T00:25:45.513 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.510+0000 7fbb0f7fe700 1 -- 172.21.15.87:0/3413164007 <== mgr.14142 v2:172.21.15.87:6800/3977108171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fbb18102150 con 0x7fbaf802ff20 2024-02-20T00:25:45.513 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-20T00:25:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 msgr2=0x7fbaf80323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 0x7fbaf80323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb080096a0 tx=0x7fbb08006b40).stop 2024-02-20T00:25:45.515 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 msgr2=0x7fbb1810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbb00006800 tx=0x7fbb00006ee0).stop 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 shutdown_connections 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fbaf802ff20 0x7fbaf80323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 --2- 172.21.15.87:0/3413164007 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb180703b0 0x7fbb1810cd80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.512+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 >> 172.21.15.87:0/3413164007 conn(0x7fbb180f6530 msgr2=0x7fbb180f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.513+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 shutdown_connections 2024-02-20T00:25:45.516 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:45.513+0000 7fbb1d946700 1 -- 172.21.15.87:0/3413164007 wait complete. 2024-02-20T00:25:45.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:44.134340+0000 mgr.smithi087.iydsqo (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:45.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:44.135003+0000 mgr.smithi087.iydsqo (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-20T00:25:45.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:44.137302+0000 mon.smithi087 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:46.128 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-20T00:25:46.716 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.710+0000 7ff8c5c87700 1 Processor -- start 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.710+0000 7ff8c5c87700 1 -- start start 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.710+0000 7ff8c5c87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.710+0000 7ff8c5c87700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c0072920 con 0x7ff8c0072500 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43084/0 (socket says 172.21.15.87:43084) 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8bf7fe700 1 -- 172.21.15.87:0/1866238478 learned_addr learned my addr 172.21.15.87:0/1866238478 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8bf7fe700 1 -- 172.21.15.87:0/1866238478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c0070f90 con 0x7ff8c0072500 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8bf7fe700 1 --2- 172.21.15.87:0/1866238478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff8b400b5f0 tx=0x7ff8b4006ee0).ready entity=mon.0 client_cookie=be846001cf679731 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8be7fc700 1 -- 172.21.15.87:0/1866238478 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b400d600 con 0x7ff8c0072500 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8be7fc700 1 -- 172.21.15.87:0/1866238478 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff8b400dbf0 con 0x7ff8c0072500 2024-02-20T00:25:46.717 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.711+0000 7ff8be7fc700 1 -- 172.21.15.87:0/1866238478 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b4011cb0 con 0x7ff8c0072500 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 -- 172.21.15.87:0/1866238478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 msgr2=0x7ff8c0070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/1866238478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff8b400b5f0 tx=0x7ff8b4006ee0).stop 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 -- 172.21.15.87:0/1866238478 shutdown_connections 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/1866238478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c0070aa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 -- 172.21.15.87:0/1866238478 >> 172.21.15.87:0/1866238478 conn(0x7ff8c00f7590 msgr2=0x7ff8c00f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 -- 172.21.15.87:0/1866238478 shutdown_connections 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.712+0000 7ff8c5c87700 1 -- 172.21.15.87:0/1866238478 wait complete. 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8c5c87700 1 Processor -- start 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8c5c87700 1 -- start start 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8c5c87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8c5c87700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c010d300 con 0x7ff8c0072500 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43086/0 (socket says 172.21.15.87:43086) 2024-02-20T00:25:46.718 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8bf7fe700 1 -- 172.21.15.87:0/2041715330 learned_addr learned my addr 172.21.15.87:0/2041715330 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:46.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.713+0000 7ff8bf7fe700 1 -- 172.21.15.87:0/2041715330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b400b200 con 0x7ff8c0072500 2024-02-20T00:25:46.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8bf7fe700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff8b400df90 tx=0x7ff8b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:46.719 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b401b410 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff8b401ba00 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c010d500 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b401aa30 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.714+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c010e110 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c0059100 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff8b400f070 con 0x7ff8c0072500 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8a7fff700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 0x7ff8ac0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:46.720 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff8b40413a0 con 0x7ff8c0072500 2024-02-20T00:25:46.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8beffd700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 0x7ff8ac0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:46.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.715+0000 7ff8beffd700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 0x7ff8ac0323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff8b00096a0 tx=0x7ff8b0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:46.721 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.719+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8b4010360 con 0x7ff8c0072500 2024-02-20T00:25:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:45.509505+0000 mgr.smithi087.iydsqo (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:25:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:25:45.510037+0000 mgr.smithi087.iydsqo (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-20T00:25:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:45.512153+0000 mon.smithi087 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:46.871 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.868+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff8c0102180 con 0x7ff8c0072500 2024-02-20T00:25:46.873 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.871+0000 7ff8a7fff700 1 -- 172.21.15.87:0/2041715330 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff8b4016030 con 0x7ff8c0072500 2024-02-20T00:25:46.876 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 msgr2=0x7ff8ac0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:46.876 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 0x7ff8ac0323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff8b00096a0 tx=0x7ff8b0006b40).stop 2024-02-20T00:25:46.876 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 msgr2=0x7ff8c010ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:46.876 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff8b400df90 tx=0x7ff8b4006ee0).stop 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 shutdown_connections 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7ff8ac02ff20 0x7ff8ac0323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 --2- 172.21.15.87:0/2041715330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff8c0072500 0x7ff8c010ce10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.872+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 >> 172.21.15.87:0/2041715330 conn(0x7ff8c00f7590 msgr2=0x7ff8c00f9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.873+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 shutdown_connections 2024-02-20T00:25:46.877 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:46.873+0000 7ff8c5c87700 1 -- 172.21.15.87:0/2041715330 wait complete. 2024-02-20T00:25:47.476 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-20T00:25:48.074 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.026+0000 7fc2ec222700 1 Processor -- start 2024-02-20T00:25:48.074 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2ec222700 1 -- start start 2024-02-20T00:25:48.074 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2ec222700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2ec222700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40fc880 con 0x7fc2e40703f0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e9fbe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e9fbe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43090/0 (socket says 172.21.15.87:43090) 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e9fbe700 1 -- 172.21.15.87:0/2294569531 learned_addr learned my addr 172.21.15.87:0/2294569531 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e9fbe700 1 -- 172.21.15.87:0/2294569531 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4101380 con 0x7fc2e40703f0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e9fbe700 1 --2- 172.21.15.87:0/2294569531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fc2e000b5f0 tx=0x7fc2e0006ee0).ready entity=mon.0 client_cookie=a2f079f227ecca47 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e8fbc700 1 -- 172.21.15.87:0/2294569531 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e000d600 con 0x7fc2e40703f0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e8fbc700 1 -- 172.21.15.87:0/2294569531 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc2e000dbf0 con 0x7fc2e40703f0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.027+0000 7fc2e8fbc700 1 -- 172.21.15.87:0/2294569531 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e0011ce0 con 0x7fc2e40703f0 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 -- 172.21.15.87:0/2294569531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 msgr2=0x7fc2e4100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2294569531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fc2e000b5f0 tx=0x7fc2e0006ee0).stop 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 -- 172.21.15.87:0/2294569531 shutdown_connections 2024-02-20T00:25:48.075 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2294569531 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e4100e90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 -- 172.21.15.87:0/2294569531 >> 172.21.15.87:0/2294569531 conn(0x7fc2e40f6530 msgr2=0x7fc2e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 -- 172.21.15.87:0/2294569531 shutdown_connections 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 -- 172.21.15.87:0/2294569531 wait complete. 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.028+0000 7fc2ec222700 1 Processor -- start 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2ec222700 1 -- start start 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2ec222700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2ec222700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e410d2b0 con 0x7fc2e40703f0 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2e9fbe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2e9fbe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43106/0 (socket says 172.21.15.87:43106) 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2e9fbe700 1 -- 172.21.15.87:0/2607277217 learned_addr learned my addr 172.21.15.87:0/2607277217 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2e9fbe700 1 -- 172.21.15.87:0/2607277217 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e000b200 con 0x7fc2e40703f0 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2e9fbe700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc2e0011ed0 tx=0x7fc2e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e001b570 con 0x7fc2e40703f0 2024-02-20T00:25:48.076 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc2e001bb60 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e410d4b0 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e001aad0 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.029+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e410dff0 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.030+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc2e000f070 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.030+0000 7fc2d27fc700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 0x7fc2cc0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.030+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4059100 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.030+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc2e0041500 con 0x7fc2e40703f0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.030+0000 7fc2e97bd700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 0x7fc2cc0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.031+0000 7fc2e97bd700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 0x7fc2cc0323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc2d40096a0 tx=0x7fc2d4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:48.077 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.034+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e0016070 con 0x7fc2e40703f0 2024-02-20T00:25:48.202 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.199+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc2e410ee20 con 0x7fc2e40703f0 2024-02-20T00:25:48.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.202+0000 7fc2d27fc700 1 -- 172.21.15.87:0/2607277217 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc2e0022ac0 con 0x7fc2e40703f0 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 msgr2=0x7fc2cc0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 0x7fc2cc0323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc2d40096a0 tx=0x7fc2d4006b40).stop 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 msgr2=0x7fc2e410cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc2e0011ed0 tx=0x7fc2e0006ee0).stop 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 shutdown_connections 2024-02-20T00:25:48.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fc2cc02ff20 0x7fc2cc0323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:48.209 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 --2- 172.21.15.87:0/2607277217 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e40703f0 0x7fc2e410cdc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:48.209 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.204+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 >> 172.21.15.87:0/2607277217 conn(0x7fc2e40f6530 msgr2=0x7fc2e40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:48.209 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.205+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 shutdown_connections 2024-02-20T00:25:48.209 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:48.205+0000 7fc2ec222700 1 -- 172.21.15.87:0/2607277217 wait complete. 2024-02-20T00:25:48.236 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:47 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:46.872661+0000 mon.smithi087 (mon.0) 82 : audit [INF] from='client.? 172.21.15.87:0/2041715330' entity='client.admin' 2024-02-20T00:25:49.443 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:48.204171+0000 mon.smithi087 (mon.0) 83 : audit [INF] from='client.? 172.21.15.87:0/2607277217' entity='client.admin' 2024-02-20T00:25:49.976 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling the dashboard module... 2024-02-20T00:25:49.976 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-20T00:25:50.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:49.464804+0000 mon.smithi087 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:50.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:49.686061+0000 mon.smithi087 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:50.807 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.701+0000 7f0f7d036700 1 Processor -- start 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.701+0000 7f0f7d036700 1 -- start start 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.701+0000 7f0f7d036700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.701+0000 7f0f7d036700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780fc7b0 con 0x7f0f780fbea0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f76d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f76d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49900/0 (socket says 172.21.15.87:49900) 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f76d9d700 1 -- 172.21.15.87:0/2029418750 learned_addr learned my addr 172.21.15.87:0/2029418750 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f76d9d700 1 -- 172.21.15.87:0/2029418750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780fc8f0 con 0x7f0f780fbea0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f76d9d700 1 --2- 172.21.15.87:0/2029418750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0f6000b5f0 tx=0x7f0f60006ee0).ready entity=mon.0 client_cookie=a2720f8ad2ef115a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f75d9b700 1 -- 172.21.15.87:0/2029418750 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6000d600 con 0x7f0f780fbea0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f75d9b700 1 -- 172.21.15.87:0/2029418750 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f6000dc30 con 0x7f0f780fbea0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.702+0000 7f0f75d9b700 1 -- 172.21.15.87:0/2029418750 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f60011d30 con 0x7f0f780fbea0 2024-02-20T00:25:50.808 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- 172.21.15.87:0/2029418750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 msgr2=0x7f0f780fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 --2- 172.21.15.87:0/2029418750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0f6000b5f0 tx=0x7f0f60006ee0).stop 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- 172.21.15.87:0/2029418750 shutdown_connections 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 --2- 172.21.15.87:0/2029418750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f780fc2c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- 172.21.15.87:0/2029418750 >> 172.21.15.87:0/2029418750 conn(0x7f0f780f76a0 msgr2=0x7f0f780f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- 172.21.15.87:0/2029418750 shutdown_connections 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- 172.21.15.87:0/2029418750 wait complete. 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 Processor -- start 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- start start 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.703+0000 7f0f7d036700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7810b0a0 con 0x7f0f780fbea0 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f76d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f76d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49908/0 (socket says 172.21.15.87:49908) 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f76d9d700 1 -- 172.21.15.87:0/419995451 learned_addr learned my addr 172.21.15.87:0/419995451 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f76d9d700 1 -- 172.21.15.87:0/419995451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6000b200 con 0x7f0f780fbea0 2024-02-20T00:25:50.809 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f76d9d700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0f60006b80 tx=0x7f0f60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6000d9e0 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f6001b8e0 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7810b2a0 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6001a870 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.704+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7810b730 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0f6000f070 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f6f7fe700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 0x7f0f58032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0f600423c0 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f7659c700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 0x7f0f58032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f78045ad0 con 0x7f0f780fbea0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.705+0000 7f0f7659c700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 0x7f0f58032430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0f680096a0 tx=0x7f0f68006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:50.810 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.709+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f60016070 con 0x7f0f780fbea0 2024-02-20T00:25:50.900 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:50.898+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0f78104bf0 con 0x7f0f780fbea0 2024-02-20T00:25:51.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.472+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0f600257e0 con 0x7f0f780fbea0 2024-02-20T00:25:51.475 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.473+0000 7f0f6f7fe700 1 -- 172.21.15.87:0/419995451 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0f6002f430 con 0x7f0f780fbea0 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 msgr2=0x7f0f58032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 0x7f0f58032430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0f680096a0 tx=0x7f0f68006b40).stop 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 msgr2=0x7f0f7810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0f60006b80 tx=0x7f0f60006ee0).stop 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 shutdown_connections 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f0f5802ff70 0x7f0f58032430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 --2- 172.21.15.87:0/419995451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f780fbea0 0x7f0f7810abb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 >> 172.21.15.87:0/419995451 conn(0x7f0f780f76a0 msgr2=0x7f0f780f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 shutdown_connections 2024-02-20T00:25:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:51.476+0000 7f0f7d036700 1 -- 172.21.15.87:0/419995451 wait complete. 2024-02-20T00:25:51.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:51 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:50.899859+0000 mon.smithi087 (mon.0) 86 : audit [INF] from='client.? 172.21.15.87:0/419995451' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T00:25:51.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:51 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:51.191439+0000 mon.smithi087 (mon.0) 87 : audit [INF] from='mgr.14142 172.21.15.87:0/126796871' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:52.723 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:51.474464+0000 mon.smithi087 (mon.0) 88 : audit [INF] from='client.? 172.21.15.87:0/419995451' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T00:25:52.724 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:51.474518+0000 mon.smithi087 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi087.iydsqo(active, since 11s) 2024-02-20T00:25:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-20T00:25:53.576 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.353+0000 7f8b72e9c700 1 Processor -- start 2024-02-20T00:25:53.576 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.353+0000 7f8b72e9c700 1 -- start start 2024-02-20T00:25:53.576 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.353+0000 7f8b72e9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:53.576 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.353+0000 7f8b72e9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0fb6a0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.576 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b70c38700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b70c38700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49924/0 (socket says 172.21.15.87:49924) 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b70c38700 1 -- 172.21.15.87:0/1189845315 learned_addr learned my addr 172.21.15.87:0/1189845315 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b70c38700 1 -- 172.21.15.87:0/1189845315 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c0fb7e0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b70c38700 1 --2- 172.21.15.87:0/1189845315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8b6000b5f0 tx=0x7f8b60006ee0).ready entity=mon.0 client_cookie=2927941126362d6d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b6b7fe700 1 -- 172.21.15.87:0/1189845315 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b6000d600 con 0x7f8b6c0fad90 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b6b7fe700 1 -- 172.21.15.87:0/1189845315 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b6000dc30 con 0x7f8b6c0fad90 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.354+0000 7f8b6b7fe700 1 -- 172.21.15.87:0/1189845315 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b60011d30 con 0x7f8b6c0fad90 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- 172.21.15.87:0/1189845315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 msgr2=0x7f8b6c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/1189845315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8b6000b5f0 tx=0x7f8b60006ee0).stop 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- 172.21.15.87:0/1189845315 shutdown_connections 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/1189845315 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c0fb1b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- 172.21.15.87:0/1189845315 >> 172.21.15.87:0/1189845315 conn(0x7f8b6c0f6530 msgr2=0x7f8b6c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- 172.21.15.87:0/1189845315 shutdown_connections 2024-02-20T00:25:53.577 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- 172.21.15.87:0/1189845315 wait complete. 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 Processor -- start 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.355+0000 7f8b72e9c700 1 -- start start 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b72e9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b72e9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c070450 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b70c38700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b70c38700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49940/0 (socket says 172.21.15.87:49940) 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b70c38700 1 -- 172.21.15.87:0/2820909924 learned_addr learned my addr 172.21.15.87:0/2820909924 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b70c38700 1 -- 172.21.15.87:0/2820909924 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6000b200 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.356+0000 7f8b70c38700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8b6000e040 tx=0x7f8b60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b6000d9e0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c070650 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b6001b8e0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b6001a880 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c10df80 con 0x7f8b6c0fad90 2024-02-20T00:25:53.578 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8b6000f070 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b697fa700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 0x7f8b50032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.357+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c045ad0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.358+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8b60042430 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.358+0000 7f8b6bfff700 1 -- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 msgr2=0x7f8b50032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.358+0000 7f8b6bfff700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 0x7f8b50032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.362+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b60016070 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.557+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8b6c0586e0 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.558+0000 7f8b697fa700 1 -- 172.21.15.87:0/2820909924 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f8b60022d70 con 0x7f8b6c0fad90 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.559+0000 7f8b6bfff700 1 -- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 msgr2=0x7f8b50032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.559+0000 7f8b6bfff700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 0x7f8b50032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 msgr2=0x7f8b50032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 0x7f8b50032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 msgr2=0x7f8b6c10d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8b6000e040 tx=0x7f8b60006ee0).stop 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 shutdown_connections 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7f8b5002ff70 0x7f8b50032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 --2- 172.21.15.87:0/2820909924 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8b6c0fad90 0x7f8b6c10d8d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 >> 172.21.15.87:0/2820909924 conn(0x7f8b6c0f6530 msgr2=0x7f8b6c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 shutdown_connections 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:53.560+0000 7f8b72e9c700 1 -- 172.21.15.87:0/2820909924 wait complete. 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "active_name": "smithi087.iydsqo", 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T00:25:53.580 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:54.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:53.558731+0000 mon.smithi087 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.87:0/2820909924' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T00:25:54.323 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for the mgr to restart... 2024-02-20T00:25:54.323 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr epoch 13... 2024-02-20T00:25:54.323 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.934+0000 7fef5dc94700 1 Processor -- start 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.934+0000 7fef5dc94700 1 -- start start 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.934+0000 7fef5dc94700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef5dc94700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580fef80 con 0x7fef580703f0 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:54.941 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49944/0 (socket says 172.21.15.87:49944) 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef577fe700 1 -- 172.21.15.87:0/3136801280 learned_addr learned my addr 172.21.15.87:0/3136801280 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef577fe700 1 -- 172.21.15.87:0/3136801280 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580ff0c0 con 0x7fef580703f0 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef577fe700 1 --2- 172.21.15.87:0/3136801280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fef4800b5f0 tx=0x7fef48006ee0).ready entity=mon.0 client_cookie=8ab8f86d4099f058 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef567fc700 1 -- 172.21.15.87:0/3136801280 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef4800d600 con 0x7fef580703f0 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef567fc700 1 -- 172.21.15.87:0/3136801280 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef4800dc30 con 0x7fef580703f0 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.935+0000 7fef567fc700 1 -- 172.21.15.87:0/3136801280 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef48011d60 con 0x7fef580703f0 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 -- 172.21.15.87:0/3136801280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 msgr2=0x7fef580fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 --2- 172.21.15.87:0/3136801280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fef4800b5f0 tx=0x7fef48006ee0).stop 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 -- 172.21.15.87:0/3136801280 shutdown_connections 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 --2- 172.21.15.87:0/3136801280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fea90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 -- 172.21.15.87:0/3136801280 >> 172.21.15.87:0/3136801280 conn(0x7fef580f6530 msgr2=0x7fef580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 -- 172.21.15.87:0/3136801280 shutdown_connections 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 -- 172.21.15.87:0/3136801280 wait complete. 2024-02-20T00:25:54.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.936+0000 7fef5dc94700 1 Processor -- start 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef5dc94700 1 -- start start 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef5dc94700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef5dc94700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580fa990 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49946/0 (socket says 172.21.15.87:49946) 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef577fe700 1 -- 172.21.15.87:0/649744170 learned_addr learned my addr 172.21.15.87:0/649744170 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef577fe700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4800b200 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.937+0000 7fef577fe700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fef4800e040 tx=0x7fef48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.938+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef48011e10 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.938+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef4801b8e0 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.938+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef580fab30 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.938+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef4801a930 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.938+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef580fafc0 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fef4800f070 con 0x7fef580703f0 2024-02-20T00:25:54.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef3ffff700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef48011610 con 0x7fef4402ffc0 2024-02-20T00:25:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fef48040e50 con 0x7fef580703f0 2024-02-20T00:25:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef56ffd700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 msgr2=0x7fef44032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:54.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:54.939+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:25:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:55.140+0000 7fef56ffd700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 msgr2=0x7fef44032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:55.140+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:25:55.543 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:55.541+0000 7fef56ffd700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 msgr2=0x7fef44032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:55.543 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:55.541+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:25:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:56.342+0000 7fef56ffd700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 msgr2=0x7fef44032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/3977108171 2024-02-20T00:25:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:56.342+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T00:25:56.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:56.520+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fef48019ce0 con 0x7fef580703f0 2024-02-20T00:25:56.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:56.520+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 msgr2=0x7fef44032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:25:56.523 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:56.520+0000 7fef3ffff700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:56.756 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:56.466560+0000 mon.smithi087 (mon.0) 91 : cluster [INF] Active manager daemon smithi087.iydsqo restarted 2024-02-20T00:25:56.756 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:56.466950+0000 mon.smithi087 (mon.0) 92 : cluster [INF] Activating manager daemon smithi087.iydsqo 2024-02-20T00:25:56.756 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:56.471115+0000 mon.smithi087 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-20T00:25:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.529+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fef4801aa90 con 0x7fef580703f0 2024-02-20T00:25:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.529+0000 7fef3ffff700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 0x7fef44035ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:57.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.529+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef48011610 con 0x7fef44033ae0 2024-02-20T00:25:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.529+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 0x7fef44035ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.530+0000 7fef56ffd700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 0x7fef44035ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fef4c0036d0 tx=0x7fef4c006d20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.530+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fef48011610 con 0x7fef44033ae0 2024-02-20T00:25:57.538 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fef58059100 con 0x7fef44033ae0 2024-02-20T00:25:57.538 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef3ffff700 1 -- 172.21.15.87:0/649744170 <== mgr.14162 v2:172.21.15.87:6800/3526433496 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fef58059100 con 0x7fef44033ae0 2024-02-20T00:25:57.538 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-20T00:25:57.538 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-20T00:25:57.539 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T00:25:57.539 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 msgr2=0x7fef44035ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 0x7fef44035ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fef4c0036d0 tx=0x7fef4c006d20).stop 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 msgr2=0x7fef580fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fef4800e040 tx=0x7fef48006ee0).stop 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 shutdown_connections 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef44033ae0 0x7fef44035ed0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef580703f0 0x7fef580fc510 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 --2- 172.21.15.87:0/649744170 >> [v2:172.21.15.87:6800/3977108171,v1:172.21.15.87:6801/3977108171] conn(0x7fef4402ffc0 0x7fef44032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.536+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 >> 172.21.15.87:0/649744170 conn(0x7fef580f6530 msgr2=0x7fef580f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.537+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 shutdown_connections 2024-02-20T00:25:57.540 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:57.537+0000 7fef5dc94700 1 -- 172.21.15.87:0/649744170 wait complete. 2024-02-20T00:25:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:56.522855+0000 mon.smithi087 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi087.iydsqo(active, starting, since 0.0559888s) 2024-02-20T00:25:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.523960+0000 mon.smithi087 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata", "id": "smithi087"}]: dispatch 2024-02-20T00:25:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.526370+0000 mon.smithi087 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mgr metadata", "who": "smithi087.iydsqo", "id": "smithi087.iydsqo"}]: dispatch 2024-02-20T00:25:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.526550+0000 mon.smithi087 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:25:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.526661+0000 mon.smithi087 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.526788+0000 mon.smithi087 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:56.541112+0000 mon.smithi087 (mon.0) 100 : cluster [INF] Manager daemon smithi087.iydsqo is now available 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.638679+0000 mon.smithi087 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.640931+0000 mon.smithi087 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.661923+0000 mon.smithi087 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:25:57.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:56.665068+0000 mon.smithi087 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:25:58.494 INFO:teuthology.orchestra.run.smithi087.stderr:mgr epoch 13 is available 2024-02-20T00:25:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:Generating a dashboard self-signed certificate... 2024-02-20T00:25:58.495 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-20T00:25:58.754 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:57.531095+0000 mgr.smithi087.iydsqo (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T00:25:58.754 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:57.531552+0000 mon.smithi087 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi087.iydsqo(active, since 1.06468s) 2024-02-20T00:25:58.754 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:57.537215+0000 mgr.smithi087.iydsqo (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T00:25:58.754 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:25:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:58.125971+0000 mon.smithi087 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f32870a1700 1 Processor -- start 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f32870a1700 1 -- start start 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f32870a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f32870a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32800fc7b0 con 0x7f32800fbea0 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f3284e3d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f3284e3d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57222/0 (socket says 172.21.15.87:57222) 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.242+0000 7f3284e3d700 1 -- 172.21.15.87:0/1350655880 learned_addr learned my addr 172.21.15.87:0/1350655880 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f3284e3d700 1 -- 172.21.15.87:0/1350655880 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32800fc8f0 con 0x7f32800fbea0 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f3284e3d700 1 --2- 172.21.15.87:0/1350655880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f327400b5f0 tx=0x7f3274006ee0).ready entity=mon.0 client_cookie=d61b6278e751ce16 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:59.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f327f7fe700 1 -- 172.21.15.87:0/1350655880 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327400d600 con 0x7f32800fbea0 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f327f7fe700 1 -- 172.21.15.87:0/1350655880 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f327400dc30 con 0x7f32800fbea0 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f327f7fe700 1 -- 172.21.15.87:0/1350655880 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274011d30 con 0x7f32800fbea0 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f32870a1700 1 -- 172.21.15.87:0/1350655880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 msgr2=0x7f32800fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f32870a1700 1 --2- 172.21.15.87:0/1350655880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f327400b5f0 tx=0x7f3274006ee0).stop 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f32870a1700 1 -- 172.21.15.87:0/1350655880 shutdown_connections 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f32870a1700 1 --2- 172.21.15.87:0/1350655880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f32800fc2c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.243+0000 7f32870a1700 1 -- 172.21.15.87:0/1350655880 >> 172.21.15.87:0/1350655880 conn(0x7f32800f76a0 msgr2=0x7f32800f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 -- 172.21.15.87:0/1350655880 shutdown_connections 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 -- 172.21.15.87:0/1350655880 wait complete. 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 Processor -- start 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 -- start start 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f32870a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328010b0f0 con 0x7f32800fbea0 2024-02-20T00:25:59.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.244+0000 7f3284e3d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:59.249 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f3284e3d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57232/0 (socket says 172.21.15.87:57232) 2024-02-20T00:25:59.249 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f3284e3d700 1 -- 172.21.15.87:0/4017524205 learned_addr learned my addr 172.21.15.87:0/4017524205 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:25:59.249 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f3284e3d700 1 -- 172.21.15.87:0/4017524205 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327400b200 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f3284e3d700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3274000c00 tx=0x7f3274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327400d9e0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f327401b8e0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328010b2f0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327401a7f0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.245+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328010b780 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f327400f070 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f327d7fa700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 0x7f3264032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32740415a0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f327ffff700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 0x7f3264032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3280045ad0 con 0x7f32800fbea0 2024-02-20T00:25:59.250 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.246+0000 7f327ffff700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 0x7f3264032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:25:59.252 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.250+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3274016070 con 0x7f32800fbea0 2024-02-20T00:25:59.410 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.408+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3280104cc0 con 0x7f326402ffc0 2024-02-20T00:25:59.499 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.497+0000 7f327d7fa700 1 -- 172.21.15.87:0/4017524205 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3280104cc0 con 0x7f326402ffc0 2024-02-20T00:25:59.499 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-20T00:25:59.501 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 msgr2=0x7f3264032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 0x7f3264032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f326c0096a0 tx=0x7f326c006b40).stop 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 msgr2=0x7f328010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3274000c00 tx=0x7f3274006ee0).stop 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 shutdown_connections 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f326402ffc0 0x7f3264032480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 --2- 172.21.15.87:0/4017524205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f32800fbea0 0x7f328010ac00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 >> 172.21.15.87:0/4017524205 conn(0x7f32800f76a0 msgr2=0x7f32800f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 shutdown_connections 2024-02-20T00:25:59.502 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:25:59.499+0000 7f32870a1700 1 -- 172.21.15.87:0/4017524205 wait complete. 2024-02-20T00:26:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial admin user... 2024-02-20T00:26:00.349 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3n3zl6y7:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-20T00:26:00.351 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:25:59.127360+0000 mon.smithi087 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi087.iydsqo(active, since 2s) 2024-02-20T00:26:00.351 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:59.409993+0000 mgr.smithi087.iydsqo (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:26:00.351 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:59.496475+0000 mon.smithi087 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:00.351 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:25:59.498266+0000 mon.smithi087 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f1478b4b700 1 Processor -- start 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f1478b4b700 1 -- start start 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f1478b4b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f1478b4b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740fb6a0 con 0x7f14740fad90 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f147259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:00.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f147259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57248/0 (socket says 172.21.15.87:57248) 2024-02-20T00:26:00.919 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.914+0000 7f147259c700 1 -- 172.21.15.87:0/1612526643 learned_addr learned my addr 172.21.15.87:0/1612526643 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:00.919 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f147259c700 1 -- 172.21.15.87:0/1612526643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740fb7e0 con 0x7f14740fad90 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f147259c700 1 --2- 172.21.15.87:0/1612526643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f146400b5f0 tx=0x7f1464006ee0).ready entity=mon.0 client_cookie=93ec4d1a2dc4faca server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f147159a700 1 -- 172.21.15.87:0/1612526643 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f146400d600 con 0x7f14740fad90 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f147159a700 1 -- 172.21.15.87:0/1612526643 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f146400dc30 con 0x7f14740fad90 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f147159a700 1 -- 172.21.15.87:0/1612526643 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1464011d30 con 0x7f14740fad90 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f1478b4b700 1 -- 172.21.15.87:0/1612526643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 msgr2=0x7f14740fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1612526643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f146400b5f0 tx=0x7f1464006ee0).stop 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.915+0000 7f1478b4b700 1 -- 172.21.15.87:0/1612526643 shutdown_connections 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1612526643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f14740fb1b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 -- 172.21.15.87:0/1612526643 >> 172.21.15.87:0/1612526643 conn(0x7f14740f6530 msgr2=0x7f14740f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 -- 172.21.15.87:0/1612526643 shutdown_connections 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 -- 172.21.15.87:0/1612526643 wait complete. 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 Processor -- start 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.916+0000 7f1478b4b700 1 -- start start 2024-02-20T00:26:00.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1478b4b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1478b4b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474109f80 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f147259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f147259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57260/0 (socket says 172.21.15.87:57260) 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f147259c700 1 -- 172.21.15.87:0/1761678775 learned_addr learned my addr 172.21.15.87:0/1761678775 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f147259c700 1 -- 172.21.15.87:0/1761678775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146400b200 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f147259c700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1464006b80 tx=0x7f1464006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f146400da60 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147410a180 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f146401b8e0 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f146401a960 con 0x7f14740fad90 2024-02-20T00:26:00.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.917+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147410a610 con 0x7f14740fad90 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.918+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f146400f070 con 0x7f14740fad90 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.918+0000 7f1462ffd700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 0x7f1458032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.918+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1464041580 con 0x7f14740fad90 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.918+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474045ad0 con 0x7f14740fad90 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.918+0000 7f1471d9b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 0x7f1458032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:00.922 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.919+0000 7f1471d9b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 0x7f1458032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f14680096a0 tx=0x7f1468006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:00.924 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:00.922+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1464016070 con 0x7f14740fad90 2024-02-20T00:26:01.092 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.090+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1474103c10 con 0x7f145802ff70 2024-02-20T00:26:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.310+0000 7f1462ffd700 1 -- 172.21.15.87:0/1761678775 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1474103c10 con 0x7f145802ff70 2024-02-20T00:26:01.312 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$uBzS1eSdi/DGdNu5.YBvsOr30Y6ud8Zax7yW4RhHVePSiKvMDZIfG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708388761, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 msgr2=0x7f1458032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 0x7f1458032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f14680096a0 tx=0x7f1468006b40).stop 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 msgr2=0x7f1474109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1464006b80 tx=0x7f1464006ee0).stop 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 shutdown_connections 2024-02-20T00:26:01.314 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f145802ff70 0x7f1458032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:01.315 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 --2- 172.21.15.87:0/1761678775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f14740fad90 0x7f1474109a90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:01.315 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 >> 172.21.15.87:0/1761678775 conn(0x7f14740f6530 msgr2=0x7f14740f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:01.315 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 shutdown_connections 2024-02-20T00:26:01.315 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:01.312+0000 7f1478b4b700 1 -- 172.21.15.87:0/1761678775 wait complete. 2024-02-20T00:26:01.934 INFO:teuthology.orchestra.run.smithi087.stderr:Fetching dashboard port number... 2024-02-20T00:26:01.934 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp9o3e1j8h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpo1hqqdgj:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-20T00:26:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4edcae6700 1 Processor -- start 2024-02-20T00:26:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4edcae6700 1 -- start start 2024-02-20T00:26:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4edcae6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:02.480 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4edcae6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80fc840 con 0x7f4ed8070430 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57264/0 (socket says 172.21.15.87:57264) 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.448+0000 7f4ed659c700 1 -- 172.21.15.87:0/599228985 learned_addr learned my addr 172.21.15.87:0/599228985 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4ed659c700 1 -- 172.21.15.87:0/599228985 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8101340 con 0x7f4ed8070430 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4ed659c700 1 --2- 172.21.15.87:0/599228985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4ecc00b5f0 tx=0x7f4ecc006ee0).ready entity=mon.0 client_cookie=db119bb155c70c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4ed559a700 1 -- 172.21.15.87:0/599228985 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc00d600 con 0x7f4ed8070430 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4ed559a700 1 -- 172.21.15.87:0/599228985 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ecc00dc30 con 0x7f4ed8070430 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4ed559a700 1 -- 172.21.15.87:0/599228985 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc011d30 con 0x7f4ed8070430 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4edcae6700 1 -- 172.21.15.87:0/599228985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 msgr2=0x7f4ed8100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.449+0000 7f4edcae6700 1 --2- 172.21.15.87:0/599228985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4ecc00b5f0 tx=0x7f4ecc006ee0).stop 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- 172.21.15.87:0/599228985 shutdown_connections 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 --2- 172.21.15.87:0/599228985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed8100e50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- 172.21.15.87:0/599228985 >> 172.21.15.87:0/599228985 conn(0x7f4ed80f6530 msgr2=0x7f4ed80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- 172.21.15.87:0/599228985 shutdown_connections 2024-02-20T00:26:02.481 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- 172.21.15.87:0/599228985 wait complete. 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 Processor -- start 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- start start 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4edcae6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed810d2c0 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57280/0 (socket says 172.21.15.87:57280) 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.450+0000 7f4ed659c700 1 -- 172.21.15.87:0/27298311 learned_addr learned my addr 172.21.15.87:0/27298311 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4ed659c700 1 -- 172.21.15.87:0/27298311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc00b200 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4ed659c700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4ecc00e040 tx=0x7f4ecc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc00da60 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ecc01b8e0 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed810d4c0 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ecc01a900 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.451+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed810e100 con 0x7f4ed8070430 2024-02-20T00:26:02.482 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4ecc00f070 con 0x7f4ed8070430 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4ec6ffd700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 0x7f4ebc032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ecc041610 con 0x7f4ed8070430 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4ed5d9b700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 0x7f4ebc032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed810d950 con 0x7f4ed8070430 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.452+0000 7f4ed5d9b700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 0x7f4ebc032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4ec80096a0 tx=0x7f4ec8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:02.483 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.456+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ecc016070 con 0x7f4ed8070430 2024-02-20T00:26:02.611 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.609+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4ed8059100 con 0x7f4ed8070430 2024-02-20T00:26:02.611 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.609+0000 7f4ec6ffd700 1 -- 172.21.15.87:0/27298311 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4ecc022d50 con 0x7f4ed8070430 2024-02-20T00:26:02.611 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 8443 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.611+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 msgr2=0x7f4ebc032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.611+0000 7f4edcae6700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 0x7f4ebc032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4ec80096a0 tx=0x7f4ec8006b40).stop 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.611+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 msgr2=0x7f4ed810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.611+0000 7f4edcae6700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4ecc00e040 tx=0x7f4ecc006ee0).stop 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 shutdown_connections 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ebc02ff70 0x7f4ebc032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 --2- 172.21.15.87:0/27298311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed8070430 0x7f4ed810cdd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 >> 172.21.15.87:0/27298311 conn(0x7f4ed80f6530 msgr2=0x7f4ed80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 shutdown_connections 2024-02-20T00:26:02.614 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-20T00:26:02.612+0000 7f4edcae6700 1 -- 172.21.15.87:0/27298311 wait complete. 2024-02-20T00:26:02.642 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:01.092249+0000 mgr.smithi087.iydsqo (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:26:02.643 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:01.310900+0000 mon.smithi087 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:03.307 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:26:03.317 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-20T00:26:03.317 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:26:03.324 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-20T00:26:03.325 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-20T00:26:03.325 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T00:26:03.328 INFO:teuthology.orchestra.run.smithi087.stderr:Ceph Dashboard is now available at: 2024-02-20T00:26:03.328 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.328 INFO:teuthology.orchestra.run.smithi087.stderr: URL: https://smithi087.front.sepia.ceph.com:8443/ 2024-02-20T00:26:03.328 INFO:teuthology.orchestra.run.smithi087.stderr: User: admin 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: Password: zjls1r11ld 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr:You can access the Ceph CLI with: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: ceph telemetry on 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr:For more information see: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-20T00:26:03.329 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:26:03.330 INFO:teuthology.orchestra.run.smithi087.stderr:Bootstrap complete. 2024-02-20T00:26:03.330 INFO:teuthology.orchestra.run.smithi087.stderr:Releasing lock 139849865869800 on /run/cephadm/6575da04-cf86-11ee-95bb-87774f69a715.lock 2024-02-20T00:26:03.330 INFO:teuthology.orchestra.run.smithi087.stderr:Lock 139849865869800 released on /run/cephadm/6575da04-cf86-11ee-95bb-87774f69a715.lock 2024-02-20T00:26:03.362 INFO:tasks.cephadm:Fetching config... 2024-02-20T00:26:03.362 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:26:03.362 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T00:26:03.421 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T00:26:03.421 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:26:03.421 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T00:26:03.477 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T00:26:03.478 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:26:03.478 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/mon.smithi087/keyring of=/dev/stdout 2024-02-20T00:26:03.545 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T00:26:03.545 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-20T00:26:03.545 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T00:26:03.603 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T00:26:03.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLL/1XQiCIsRkr9P+51P6XzgH6FvwBZE4rPDOmxNa+uBHyNG1NTmgKSuMXSEXqgDN4amddvN6dP6EKSxx+lcMeOBSHY9Wk/eYK5f7HtPAU4zdt42+cMgB2pIHPMRoJd39OcZVnXBSaLKadUddpEfpEhMSna0oWVCwTXuospGtihc0SL4N3fnz5Bx3jhDAtECODftMbzJAqu7XJyge+vWCkNRKZCZos+05fgMQznXppqKyKl14HoLBTg50P9Eya6+rK/n+eIPG+HK0zIIviohAA4vnC9jderiJlfYb89dlHDIuMGhPAdagn6jhWzSc9O8i863dy+tXLhxIwXPXteCXH3hKVOLI70mmcfgrsIAbG5/J46n+AXXn3j26GrwEKeDXhDl2tgBaNYLQv6qh0FFU2LXVpMWiJyPmyMWM2R/sJOiB3q6laaTNijoE+x8+uVHZTItNBYzMMs6luEriftsowZrJOy8VSy4YLb6IKYsgGd7fSwiG1eTe2Qmz4kCLuKqs= ceph-6575da04-cf86-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T00:26:03.634 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:03 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:02.610462+0000 mon.smithi087 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.87:0/27298311' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T00:26:03.674 INFO:teuthology.orchestra.run.smithi087.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLL/1XQiCIsRkr9P+51P6XzgH6FvwBZE4rPDOmxNa+uBHyNG1NTmgKSuMXSEXqgDN4amddvN6dP6EKSxx+lcMeOBSHY9Wk/eYK5f7HtPAU4zdt42+cMgB2pIHPMRoJd39OcZVnXBSaLKadUddpEfpEhMSna0oWVCwTXuospGtihc0SL4N3fnz5Bx3jhDAtECODftMbzJAqu7XJyge+vWCkNRKZCZos+05fgMQznXppqKyKl14HoLBTg50P9Eya6+rK/n+eIPG+HK0zIIviohAA4vnC9jderiJlfYb89dlHDIuMGhPAdagn6jhWzSc9O8i863dy+tXLhxIwXPXteCXH3hKVOLI70mmcfgrsIAbG5/J46n+AXXn3j26GrwEKeDXhDl2tgBaNYLQv6qh0FFU2LXVpMWiJyPmyMWM2R/sJOiB3q6laaTNijoE+x8+uVHZTItNBYzMMs6luEriftsowZrJOy8VSy4YLb6IKYsgGd7fSwiG1eTe2Qmz4kCLuKqs= ceph-6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:26:03.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLL/1XQiCIsRkr9P+51P6XzgH6FvwBZE4rPDOmxNa+uBHyNG1NTmgKSuMXSEXqgDN4amddvN6dP6EKSxx+lcMeOBSHY9Wk/eYK5f7HtPAU4zdt42+cMgB2pIHPMRoJd39OcZVnXBSaLKadUddpEfpEhMSna0oWVCwTXuospGtihc0SL4N3fnz5Bx3jhDAtECODftMbzJAqu7XJyge+vWCkNRKZCZos+05fgMQznXppqKyKl14HoLBTg50P9Eya6+rK/n+eIPG+HK0zIIviohAA4vnC9jderiJlfYb89dlHDIuMGhPAdagn6jhWzSc9O8i863dy+tXLhxIwXPXteCXH3hKVOLI70mmcfgrsIAbG5/J46n+AXXn3j26GrwEKeDXhDl2tgBaNYLQv6qh0FFU2LXVpMWiJyPmyMWM2R/sJOiB3q6laaTNijoE+x8+uVHZTItNBYzMMs6luEriftsowZrJOy8VSy4YLb6IKYsgGd7fSwiG1eTe2Qmz4kCLuKqs= ceph-6575da04-cf86-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T00:26:03.730 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLL/1XQiCIsRkr9P+51P6XzgH6FvwBZE4rPDOmxNa+uBHyNG1NTmgKSuMXSEXqgDN4amddvN6dP6EKSxx+lcMeOBSHY9Wk/eYK5f7HtPAU4zdt42+cMgB2pIHPMRoJd39OcZVnXBSaLKadUddpEfpEhMSna0oWVCwTXuospGtihc0SL4N3fnz5Bx3jhDAtECODftMbzJAqu7XJyge+vWCkNRKZCZos+05fgMQznXppqKyKl14HoLBTg50P9Eya6+rK/n+eIPG+HK0zIIviohAA4vnC9jderiJlfYb89dlHDIuMGhPAdagn6jhWzSc9O8i863dy+tXLhxIwXPXteCXH3hKVOLI70mmcfgrsIAbG5/J46n+AXXn3j26GrwEKeDXhDl2tgBaNYLQv6qh0FFU2LXVpMWiJyPmyMWM2R/sJOiB3q6laaTNijoE+x8+uVHZTItNBYzMMs6luEriftsowZrJOy8VSy4YLb6IKYsgGd7fSwiG1eTe2Qmz4kCLuKqs= ceph-6575da04-cf86-11ee-95bb-87774f69a715 2024-02-20T00:26:03.742 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T00:26:04.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:03.313263+0000 mon.smithi087 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi087.iydsqo(active, since 6s) 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 -- 172.21.15.87:0/2250805757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 msgr2=0x7f149c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 --2- 172.21.15.87:0/2250805757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c0f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f148c009fb0 tx=0x7f148c009b30).stop 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 -- 172.21.15.87:0/2250805757 shutdown_connections 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 --2- 172.21.15.87:0/2250805757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c0f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 -- 172.21.15.87:0/2250805757 >> 172.21.15.87:0/2250805757 conn(0x7f149c0f1c70 msgr2=0x7f149c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 -- 172.21.15.87:0/2250805757 shutdown_connections 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.850+0000 7f14a098c700 1 -- 172.21.15.87:0/2250805757 wait complete. 2024-02-20T00:26:06.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f14a098c700 1 Processor -- start 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f14a098c700 1 -- start start 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f14a098c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f14a098c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149c106cd0 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f149a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f149a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57298/0 (socket says 172.21.15.87:57298) 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f149a59c700 1 -- 172.21.15.87:0/1418128289 learned_addr learned my addr 172.21.15.87:0/1418128289 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.851+0000 7f149a59c700 1 -- 172.21.15.87:0/1418128289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c005040 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f149a59c700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f148c005010 tx=0x7f148c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f148c009320 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f148c013380 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f148c014e50 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149c106ed0 con 0x7f149c0f6510 2024-02-20T00:26:06.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c107360 con 0x7f149c0f6510 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.852+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f148c0263a0 con 0x7f149c0f6510 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.853+0000 7f148affd700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 0x7f14840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.853+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f148c040f10 con 0x7f149c0f6510 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.853+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149c045ad0 con 0x7f149c0f6510 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.853+0000 7f1499d9b700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 0x7f14840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.853+0000 7f1499d9b700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 0x7f14840323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f14900096a0 tx=0x7f1490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:06.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:06.857+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f148c012920 con 0x7f149c0f6510 2024-02-20T00:26:07.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.041+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f149c108110 con 0x7f149c0f6510 2024-02-20T00:26:07.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.044+0000 7f148affd700 1 -- 172.21.15.87:0/1418128289 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f148c02b020 con 0x7f149c0f6510 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 msgr2=0x7f14840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 0x7f14840323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f14900096a0 tx=0x7f1490006b40).stop 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 msgr2=0x7f149c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f148c005010 tx=0x7f148c00b040).stop 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 shutdown_connections 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f148402ff10 0x7f14840323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 --2- 172.21.15.87:0/1418128289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f149c0f6510 0x7f149c1067e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:07.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 >> 172.21.15.87:0/1418128289 conn(0x7f149c0f1c70 msgr2=0x7f149c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:07.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 shutdown_connections 2024-02-20T00:26:07.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:07.046+0000 7f14a098c700 1 -- 172.21.15.87:0/1418128289 wait complete. 2024-02-20T00:26:07.730 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:06.405431+0000 mon.smithi087 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:07.730 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:06.935059+0000 mon.smithi087 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:07.730 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:07.045228+0000 mon.smithi087 (mon.0) 115 : audit [INF] from='client.? 172.21.15.87:0/1418128289' entity='client.admin' 2024-02-20T00:26:07.838 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-02-20T00:26:07.839 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:26:07.839 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-02-20T00:26:07.860 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-02-20T00:26:07.860 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T00:26:07.921 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-02-20T00:26:07.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph orch host add smithi195 2024-02-20T00:26:08.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:26:06.941374+0000 mgr.smithi087.iydsqo (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi087 on smithi087 2024-02-20T00:26:17.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:17 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:16.526947+0000 mgr.smithi087.iydsqo (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:20.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:18.527301+0000 mgr.smithi087.iydsqo (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:20.527625+0000 mgr.smithi087.iydsqo (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:24.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:22.528029+0000 mgr.smithi087.iydsqo (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:26.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:24.528396+0000 mgr.smithi087.iydsqo (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:28.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:26.528681+0000 mgr.smithi087.iydsqo (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 -- 172.21.15.195:0/2061659334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 msgr2=0x7f77840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 --2- 172.21.15.195:0/2061659334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f77840f90f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f7778009fb0 tx=0x7f7778009b30).stop 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 -- 172.21.15.195:0/2061659334 shutdown_connections 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 --2- 172.21.15.195:0/2061659334 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f77840f90f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 -- 172.21.15.195:0/2061659334 >> 172.21.15.195:0/2061659334 conn(0x7f77840f0b70 msgr2=0x7f77840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:30.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 -- 172.21.15.195:0/2061659334 shutdown_connections 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.108+0000 7f778a252700 1 -- 172.21.15.195:0/2061659334 wait complete. 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f778a252700 1 Processor -- start 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f778a252700 1 -- start start 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f778a252700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f778a252700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784106c30 con 0x7f7784060280 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f77837fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f77837fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57030/0 (socket says 172.21.15.195:57030) 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.109+0000 7f77837fe700 1 -- 172.21.15.195:0/1599860533 learned_addr learned my addr 172.21.15.195:0/1599860533 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f77837fe700 1 -- 172.21.15.195:0/1599860533 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778005040 con 0x7f7784060280 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f77837fe700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f77780089f0 tx=0x7f777800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7778009320 con 0x7f7784060280 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7784106e30 con 0x7f7784060280 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7778013380 con 0x7f7784060280 2024-02-20T00:26:30.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77841072c0 con 0x7f7784060280 2024-02-20T00:26:30.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.110+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7778014e20 con 0x7f7784060280 2024-02-20T00:26:30.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77780263a0 con 0x7f7784060280 2024-02-20T00:26:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7784045ad0 con 0x7f7784060280 2024-02-20T00:26:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f776bfff700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 0x7f776c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7778040fa0 con 0x7f7784060280 2024-02-20T00:26:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f7782ffd700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 0x7f776c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:30.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.111+0000 7f7782ffd700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 0x7f776c0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f77740096a0 tx=0x7f7774006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:30.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.116+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77780126f0 con 0x7f7784060280 2024-02-20T00:26:30.289 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.287+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}) v1 -- 0x7f77840f1d00 con 0x7f776c02ff10 2024-02-20T00:26:30.326 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:28.528956+0000 mgr.smithi087.iydsqo (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:30.884 INFO:teuthology.orchestra.run.smithi195.stdout:Added host 'smithi195' 2024-02-20T00:26:30.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.881+0000 7f776bfff700 1 -- 172.21.15.195:0/1599860533 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f77840f1d00 con 0x7f776c02ff10 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 msgr2=0x7f776c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 0x7f776c0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f77740096a0 tx=0x7f7774006b40).stop 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 msgr2=0x7f7784106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f77780089f0 tx=0x7f777800b040).stop 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 shutdown_connections 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f776c02ff10 0x7f776c0323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 --2- 172.21.15.195:0/1599860533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7784060280 0x7f7784106740 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 >> 172.21.15.195:0/1599860533 conn(0x7f77840f0b70 msgr2=0x7f77840f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:30.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 shutdown_connections 2024-02-20T00:26:30.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:30.884+0000 7f778a252700 1 -- 172.21.15.195:0/1599860533 wait complete. 2024-02-20T00:26:31.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T00:26:31.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:30.289382+0000 mgr.smithi087.iydsqo (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:26:31.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:30.881875+0000 mon.smithi087 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:32.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:30.529379+0000 mgr.smithi087.iydsqo (mgr.14162) 14 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:32.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:26:30.882261+0000 mgr.smithi087.iydsqo (mgr.14162) 15 : cephadm [INF] Added host smithi195 2024-02-20T00:26:33.344 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- 172.21.15.195:0/3980968558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 msgr2=0x7fa1340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:33.344 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 --2- 172.21.15.195:0/3980968558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1340f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa128009fb0 tx=0x7fa128009b30).stop 2024-02-20T00:26:33.344 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- 172.21.15.195:0/3980968558 shutdown_connections 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 --2- 172.21.15.195:0/3980968558 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1340f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- 172.21.15.195:0/3980968558 >> 172.21.15.195:0/3980968558 conn(0x7fa1340f0b70 msgr2=0x7fa1340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- 172.21.15.195:0/3980968558 shutdown_connections 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- 172.21.15.195:0/3980968558 wait complete. 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 Processor -- start 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.336+0000 7fa13b24f700 1 -- start start 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa13b24f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa13b24f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa134060240 con 0x7fa1340f5410 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa138feb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa138feb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57042/0 (socket says 172.21.15.195:57042) 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa138feb700 1 -- 172.21.15.195:0/1870553272 learned_addr learned my addr 172.21.15.195:0/1870553272 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.337+0000 7fa138feb700 1 -- 172.21.15.195:0/1870553272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128005040 con 0x7fa1340f5410 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa138feb700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa128005010 tx=0x7fa12800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:33.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa128009320 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa134060440 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa128013380 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa128014e50 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa134109b90 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.338+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa128014420 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.339+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa134045ad0 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.339+0000 7fa1317fa700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 0x7fa1240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.339+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa128040f40 con 0x7fa1340f5410 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.339+0000 7fa133fff700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 0x7fa1240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:33.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.339+0000 7fa133fff700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 0x7fa1240323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:33.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.343+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1280095f0 con 0x7fa1340f5410 2024-02-20T00:26:33.516 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.514+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1340f1d20 con 0x7fa12402ff10 2024-02-20T00:26:33.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.516+0000 7fa1317fa700 1 -- 172.21.15.195:0/1870553272 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fa1340f1d20 con 0x7fa12402ff10 2024-02-20T00:26:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:33.518 INFO:teuthology.orchestra.run.smithi195.stdout:[{"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "smithi195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 msgr2=0x7fa1240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 0x7fa1240323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).stop 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 msgr2=0x7fa1341094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa128005010 tx=0x7fa12800b040).stop 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 shutdown_connections 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa12402ff10 0x7fa1240323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 --2- 172.21.15.195:0/1870553272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1340f5410 0x7fa1341094e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 >> 172.21.15.195:0/1870553272 conn(0x7fa1340f0b70 msgr2=0x7fa1340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 shutdown_connections 2024-02-20T00:26:33.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:33.517+0000 7fa13b24f700 1 -- 172.21.15.195:0/1870553272 wait complete. 2024-02-20T00:26:34.146 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T00:26:34.147 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T00:26:34.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:32.529687+0000 mgr.smithi087.iydsqo (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:34.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:33.516687+0000 mgr.smithi087.iydsqo (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 -- 172.21.15.87:0/184574715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 msgr2=0x7fe15c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 --2- 172.21.15.87:0/184574715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c0f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe150009fb0 tx=0x7fe150009b30).stop 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 -- 172.21.15.87:0/184574715 shutdown_connections 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 --2- 172.21.15.87:0/184574715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c0f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 -- 172.21.15.87:0/184574715 >> 172.21.15.87:0/184574715 conn(0x7fe15c0f0b70 msgr2=0x7fe15c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 -- 172.21.15.87:0/184574715 shutdown_connections 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.847+0000 7fe162152700 1 -- 172.21.15.87:0/184574715 wait complete. 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe162152700 1 Processor -- start 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe162152700 1 -- start start 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe162152700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe162152700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c106ca0 con 0x7fe15c0f5410 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe15b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe15b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48298/0 (socket says 172.21.15.87:48298) 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe15b7fe700 1 -- 172.21.15.87:0/3141767502 learned_addr learned my addr 172.21.15.87:0/3141767502 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe15b7fe700 1 -- 172.21.15.87:0/3141767502 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe150005040 con 0x7fe15c0f5410 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.848+0000 7fe15b7fe700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe150005010 tx=0x7fe15000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe150009320 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c106ea0 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe150013380 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe150014e50 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c107330 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.849+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe150014420 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.850+0000 7fe143fff700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 0x7fe1440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.850+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c045ad0 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.850+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe150040e30 con 0x7fe15c0f5410 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.850+0000 7fe15affd700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 0x7fe1440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.850+0000 7fe15affd700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 0x7fe1440323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe14c0096a0 tx=0x7fe14c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:35.854+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1500095f0 con 0x7fe15c0f5410 2024-02-20T00:26:36.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.008+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe15c0fb090 con 0x7fe15c0f5410 2024-02-20T00:26:36.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.070+0000 7fe143fff700 1 -- 172.21.15.87:0/3141767502 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe15002b030 con 0x7fe15c0f5410 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 msgr2=0x7fe1440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 0x7fe1440323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe14c0096a0 tx=0x7fe14c006b40).stop 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 msgr2=0x7fe15c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe150005010 tx=0x7fe15000b040).stop 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 shutdown_connections 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe14402ff10 0x7fe1440323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:36.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 --2- 172.21.15.87:0/3141767502 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe15c0f5410 0x7fe15c1067b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:36.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 >> 172.21.15.87:0/3141767502 conn(0x7fe15c0f0b70 msgr2=0x7fe15c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:36.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 shutdown_connections 2024-02-20T00:26:36.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-20T00:26:36.072+0000 7fe162152700 1 -- 172.21.15.87:0/3141767502 wait complete. 2024-02-20T00:26:36.075 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-02-20T00:26:36.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:34.530007+0000 mgr.smithi087.iydsqo (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:36.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:36.009477+0000 mon.smithi087 (mon.0) 117 : audit [INF] from='client.? 172.21.15.87:0/3141767502' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T00:26:36.636 INFO:tasks.cephadm:Adding mon.smithi087 on smithi087 2024-02-20T00:26:36.636 INFO:tasks.cephadm:Adding mon.smithi195 on smithi195 2024-02-20T00:26:36.637 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi087:172.21.15.87=smithi087;smithi195:172.21.15.195=smithi195' 2024-02-20T00:26:37.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:36.071454+0000 mon.smithi087 (mon.0) 118 : audit [INF] from='client.? 172.21.15.87:0/3141767502' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T00:26:37.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:36.071492+0000 mon.smithi087 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-20T00:26:38.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 -- 172.21.15.195:0/1988203893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 msgr2=0x7fe2280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:38.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 --2- 172.21.15.195:0/1988203893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2280f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe21c009fb0 tx=0x7fe21c009b30).stop 2024-02-20T00:26:38.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 -- 172.21.15.195:0/1988203893 shutdown_connections 2024-02-20T00:26:38.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 --2- 172.21.15.195:0/1988203893 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2280f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 -- 172.21.15.195:0/1988203893 >> 172.21.15.195:0/1988203893 conn(0x7fe2280f1c70 msgr2=0x7fe2280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 -- 172.21.15.195:0/1988203893 shutdown_connections 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.446+0000 7fe22faed700 1 -- 172.21.15.195:0/1988203893 wait complete. 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22faed700 1 Processor -- start 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22faed700 1 -- start start 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22faed700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22faed700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe228106cd0 con 0x7fe2280f6510 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22d889700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22d889700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57374/0 (socket says 172.21.15.195:57374) 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.447+0000 7fe22d889700 1 -- 172.21.15.195:0/1772972252 learned_addr learned my addr 172.21.15.195:0/1772972252 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe22d889700 1 -- 172.21.15.195:0/1772972252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c005040 con 0x7fe2280f6510 2024-02-20T00:26:38.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe22d889700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe21c005010 tx=0x7fe21c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:38.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21c009320 con 0x7fe2280f6510 2024-02-20T00:26:38.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe228106ed0 con 0x7fe2280f6510 2024-02-20T00:26:38.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe21c013380 con 0x7fe2280f6510 2024-02-20T00:26:38.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe21c014e50 con 0x7fe2280f6510 2024-02-20T00:26:38.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.448+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe228107360 con 0x7fe2280f6510 2024-02-20T00:26:38.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe21c0263a0 con 0x7fe2280f6510 2024-02-20T00:26:38.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe21a7fc700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 0x7fe2140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:38.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe21c041270 con 0x7fe2280f6510 2024-02-20T00:26:38.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe228045ad0 con 0x7fe2280f6510 2024-02-20T00:26:38.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe22d088700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 0x7fe2140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:38.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.449+0000 7fe22d088700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 0x7fe2140323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe2240096a0 tx=0x7fe224006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:38.455 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.454+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe21c012920 con 0x7fe2280f6510 2024-02-20T00:26:38.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:36.530319+0000 mgr.smithi087.iydsqo (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:38.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.627+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 --> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi087:172.21.15.87=smithi087;smithi195:172.21.15.195=smithi195", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2280f2ed0 con 0x7fe21402ff10 2024-02-20T00:26:38.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.633+0000 7fe21a7fc700 1 -- 172.21.15.195:0/1772972252 <== mgr.14162 v2:172.21.15.87:6800/3526433496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe2280f2ed0 con 0x7fe21402ff10 2024-02-20T00:26:38.634 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 msgr2=0x7fe2140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 0x7fe2140323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe2240096a0 tx=0x7fe224006b40).stop 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 msgr2=0x7fe2281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe21c005010 tx=0x7fe21c00b040).stop 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 shutdown_connections 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe21402ff10 0x7fe2140323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:38.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 --2- 172.21.15.195:0/1772972252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe2280f6510 0x7fe2281067e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:38.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 >> 172.21.15.195:0/1772972252 conn(0x7fe2280f1c70 msgr2=0x7fe2280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:38.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 shutdown_connections 2024-02-20T00:26:38.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:38.634+0000 7fe22faed700 1 -- 172.21.15.195:0/1772972252 wait complete. 2024-02-20T00:26:39.301 DEBUG:teuthology.orchestra.run.smithi195:mon.smithi195> sudo journalctl -f -n 0 -u ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195.service 2024-02-20T00:26:39.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:26:39.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:26:39.332 INFO:journalctl@ceph.mon.smithi195.smithi195.stdout:-- Logs begin at Tue 2024-02-20 00:14:57 UTC. -- 2024-02-20T00:26:40.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:39 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:38.530658+0000 mgr.smithi087.iydsqo (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:40.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:39 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:38.629118+0000 mgr.smithi087.iydsqo (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi087:172.21.15.87=smithi087;smithi195:172.21.15.195=smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:26:40.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:39 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:26:38.630517+0000 mgr.smithi087.iydsqo (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi087:172.21.15.87=smithi087;smithi195:172.21.15.195=smithi195;count:2 2024-02-20T00:26:40.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:39 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:38.633115+0000 mon.smithi087 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- 172.21.15.195:0/1635919638 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 msgr2=0x7f01e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/1635919638 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e00f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f01c8009fb0 tx=0x7f01c8009b30).stop 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- 172.21.15.195:0/1635919638 shutdown_connections 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/1635919638 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e00f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- 172.21.15.195:0/1635919638 >> 172.21.15.195:0/1635919638 conn(0x7f01e00f0b70 msgr2=0x7f01e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- 172.21.15.195:0/1635919638 shutdown_connections 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- 172.21.15.195:0/1635919638 wait complete. 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 Processor -- start 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.626+0000 7f01e4ef7700 1 -- start start 2024-02-20T00:26:41.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01e4ef7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01e4ef7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0105c10 con 0x7f01e00f5410 2024-02-20T00:26:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01de59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01de59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57394/0 (socket says 172.21.15.195:57394) 2024-02-20T00:26:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01de59c700 1 -- 172.21.15.195:0/4009212107 learned_addr learned my addr 172.21.15.195:0/4009212107 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01de59c700 1 -- 172.21.15.195:0/4009212107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8005040 con 0x7f01e00f5410 2024-02-20T00:26:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.627+0000 7f01de59c700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f01c8005010 tx=0x7f01c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c8009320 con 0x7f01e00f5410 2024-02-20T00:26:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01c8013380 con 0x7f01e00f5410 2024-02-20T00:26:41.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e0105e10 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c8014e50 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e01062a0 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01c8014420 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e0045ad0 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.628+0000 7f01d6ffd700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.629+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01c8040e30 con 0x7f01e00f5410 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.629+0000 7f01ddd9b700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:41.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.629+0000 7f01ddd9b700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 0x7f01cc0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f01d00096a0 tx=0x7f01d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.633+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01c8026bf0 con 0x7f01e00f5410 2024-02-20T00:26:41.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.836+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e0107150 con 0x7f01e00f5410 2024-02-20T00:26:41.839 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.837+0000 7f01d6ffd700 1 -- 172.21.15.195:0/4009212107 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01c801db70 con 0x7f01e00f5410 2024-02-20T00:26:41.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:41.839 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:26:41.840 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 msgr2=0x7f01cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:41.840 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 0x7f01cc0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f01d00096a0 tx=0x7f01d0006b40).stop 2024-02-20T00:26:41.840 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 msgr2=0x7f01e0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f01c8005010 tx=0x7f01c800b040).stop 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 shutdown_connections 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 --2- 172.21.15.195:0/4009212107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01e00f5410 0x7f01e0105720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 >> 172.21.15.195:0/4009212107 conn(0x7f01e00f0b70 msgr2=0x7f01e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 shutdown_connections 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:41.838+0000 7f01e4ef7700 1 -- 172.21.15.195:0/4009212107 wait complete. 2024-02-20T00:26:41.841 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:26:42.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:40.530965+0000 mgr.smithi087.iydsqo (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:43.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:41.838299+0000 mon.smithi087 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.195:0/4009212107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:26:43.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:26:43.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:26:44.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:42.531188+0000 mgr.smithi087.iydsqo (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:45.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- 172.21.15.195:0/2325619341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 msgr2=0x7fdad80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:45.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 --2- 172.21.15.195:0/2325619341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad80f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fdacc009fb0 tx=0x7fdacc009b30).stop 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- 172.21.15.195:0/2325619341 shutdown_connections 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 --2- 172.21.15.195:0/2325619341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad80f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- 172.21.15.195:0/2325619341 >> 172.21.15.195:0/2325619341 conn(0x7fdad80f0b70 msgr2=0x7fdad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- 172.21.15.195:0/2325619341 shutdown_connections 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- 172.21.15.195:0/2325619341 wait complete. 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 Processor -- start 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.310+0000 7fdadeee6700 1 -- start start 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadeee6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadeee6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad8108ee0 con 0x7fdad80f5410 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadcc82700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadcc82700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57412/0 (socket says 172.21.15.195:57412) 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadcc82700 1 -- 172.21.15.195:0/938202450 learned_addr learned my addr 172.21.15.195:0/938202450 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:45.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadcc82700 1 -- 172.21.15.195:0/938202450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc005040 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadcc82700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fdacc005010 tx=0x7fdacc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdacc009320 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.311+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad81090e0 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad8109570 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdacc013380 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdacc014e50 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad8045ad0 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdacc0095f0 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdad57fa700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 0x7fdac80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.312+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdacc040fe0 con 0x7fdad80f5410 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.313+0000 7fdad7fff700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 0x7fdac80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.313+0000 7fdad7fff700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 0x7fdac80323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdac40096a0 tx=0x7fdac4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:45.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.318+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdacc014420 con 0x7fdad80f5410 2024-02-20T00:26:45.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.522+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdad80fc0d0 con 0x7fdad80f5410 2024-02-20T00:26:45.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.522+0000 7fdad57fa700 1 -- 172.21.15.195:0/938202450 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdacc010230 con 0x7fdad80f5410 2024-02-20T00:26:45.524 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:45.525 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 msgr2=0x7fdac80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 0x7fdac80323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdac40096a0 tx=0x7fdac4006b40).stop 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 msgr2=0x7fdad81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fdacc005010 tx=0x7fdacc00b040).stop 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 shutdown_connections 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdac802ff10 0x7fdac80323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 --2- 172.21.15.195:0/938202450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdad80f5410 0x7fdad81089f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 >> 172.21.15.195:0/938202450 conn(0x7fdad80f0b70 msgr2=0x7fdad80f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 shutdown_connections 2024-02-20T00:26:45.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:45.524+0000 7fdadeee6700 1 -- 172.21.15.195:0/938202450 wait complete. 2024-02-20T00:26:45.527 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:26:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:44.531482+0000 mgr.smithi087.iydsqo (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:46.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:45.523462+0000 mon.smithi087 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.195:0/938202450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:26:47.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:26:47.214 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:26:48.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:46.531716+0000 mgr.smithi087.iydsqo (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:48.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 -- 172.21.15.195:0/3951112678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 msgr2=0x7f6a540f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/3951112678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a540f68a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6a50009fb0 tx=0x7f6a50009b30).stop 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 -- 172.21.15.195:0/3951112678 shutdown_connections 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/3951112678 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a540f68a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 -- 172.21.15.195:0/3951112678 >> 172.21.15.195:0/3951112678 conn(0x7f6a540f1c00 msgr2=0x7f6a540f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 -- 172.21.15.195:0/3951112678 shutdown_connections 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 -- 172.21.15.195:0/3951112678 wait complete. 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.976+0000 7f6a5be5c700 1 Processor -- start 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a5be5c700 1 -- start start 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a5be5c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a5be5c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54106c90 con 0x7f6a540f6480 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a59bf8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:48.979 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a59bf8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35864/0 (socket says 172.21.15.195:35864) 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a59bf8700 1 -- 172.21.15.195:0/46829547 learned_addr learned my addr 172.21.15.195:0/46829547 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a59bf8700 1 -- 172.21.15.195:0/46829547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a50005040 con 0x7f6a540f6480 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a59bf8700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6a50005010 tx=0x7f6a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a50009320 con 0x7f6a540f6480 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a54106e90 con 0x7f6a540f6480 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a50013380 con 0x7f6a540f6480 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.977+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a50014e50 con 0x7f6a540f6480 2024-02-20T00:26:48.980 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.978+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a54107320 con 0x7f6a540f6480 2024-02-20T00:26:48.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.978+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a500263a0 con 0x7f6a540f6480 2024-02-20T00:26:48.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.978+0000 7f6a4a7fc700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 0x7f6a400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:48.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.978+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a50040e90 con 0x7f6a540f6480 2024-02-20T00:26:48.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.978+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54045ad0 con 0x7f6a540f6480 2024-02-20T00:26:48.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.979+0000 7f6a593f7700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 0x7f6a400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:48.982 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.979+0000 7f6a593f7700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 0x7f6a400323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6a440096a0 tx=0x7f6a44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:48.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:48.983+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a50012920 con 0x7f6a540f6480 2024-02-20T00:26:49.189 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.189+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a54108040 con 0x7f6a540f6480 2024-02-20T00:26:49.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.189+0000 7f6a4a7fc700 1 -- 172.21.15.195:0/46829547 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a50012330 con 0x7f6a540f6480 2024-02-20T00:26:49.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:49.191 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 msgr2=0x7f6a400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 0x7f6a400323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6a440096a0 tx=0x7f6a44006b40).stop 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 msgr2=0x7f6a541067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6a50005010 tx=0x7f6a5000b040).stop 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 shutdown_connections 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6a4002ff10 0x7f6a400323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 --2- 172.21.15.195:0/46829547 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540f6480 0x7f6a541067a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 >> 172.21.15.195:0/46829547 conn(0x7f6a540f1c00 msgr2=0x7f6a540f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 shutdown_connections 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:49.191+0000 7f6a5be5c700 1 -- 172.21.15.195:0/46829547 wait complete. 2024-02-20T00:26:49.193 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:26:50.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:48.532055+0000 mgr.smithi087.iydsqo (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:50.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:49.189566+0000 mon.smithi087 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.195:0/46829547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:26:50.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:26:50.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:26:52.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:50.532308+0000 mgr.smithi087.iydsqo (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:53.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 -- 172.21.15.195:0/582323737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 msgr2=0x7f8fe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:53.974 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 --2- 172.21.15.195:0/582323737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8fd8009fb0 tx=0x7f8fd8009b30).stop 2024-02-20T00:26:53.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 -- 172.21.15.195:0/582323737 shutdown_connections 2024-02-20T00:26:53.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 --2- 172.21.15.195:0/582323737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:53.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 -- 172.21.15.195:0/582323737 >> 172.21.15.195:0/582323737 conn(0x7f8fe40f0b70 msgr2=0x7f8fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:53.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 -- 172.21.15.195:0/582323737 shutdown_connections 2024-02-20T00:26:53.975 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.972+0000 7f8fe9422700 1 -- 172.21.15.195:0/582323737 wait complete. 2024-02-20T00:26:53.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe9422700 1 Processor -- start 2024-02-20T00:26:53.976 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe9422700 1 -- start start 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe9422700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe9422700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40fb960 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35882/0 (socket says 172.21.15.195:35882) 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.973+0000 7f8fe2ffd700 1 -- 172.21.15.195:0/2447431672 learned_addr learned my addr 172.21.15.195:0/2447431672 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fe2ffd700 1 -- 172.21.15.195:0/2447431672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd8005040 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fe2ffd700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8fd80089f0 tx=0x7f8fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd8009320 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe40fbb00 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fd8013380 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd801ed80 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.974+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe40fbf90 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fd801e420 con 0x7f8fe40f5410 2024-02-20T00:26:53.977 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4059100 con 0x7f8fe40f5410 2024-02-20T00:26:53.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fcb7fe700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:53.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fd8041060 con 0x7f8fe40f5410 2024-02-20T00:26:53.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fe27fc700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:53.978 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.975+0000 7f8fe27fc700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 0x7f8fcc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8fd40096a0 tx=0x7f8fd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:53.981 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:53.980+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fd8014cc0 con 0x7f8fe40f5410 2024-02-20T00:26:54.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.186+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fe40fafa0 con 0x7f8fe40f5410 2024-02-20T00:26:54.189 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.187+0000 7f8fcb7fe700 1 -- 172.21.15.195:0/2447431672 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fd802c2d0 con 0x7f8fe40f5410 2024-02-20T00:26:54.189 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:54.189 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 msgr2=0x7f8fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 0x7f8fcc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8fd40096a0 tx=0x7f8fd4006b40).stop 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 msgr2=0x7f8fe40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f8fd80089f0 tx=0x7f8fd800b040).stop 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 shutdown_connections 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 --2- 172.21.15.195:0/2447431672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fe40f5410 0x7f8fe40fd4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 >> 172.21.15.195:0/2447431672 conn(0x7f8fe40f0b70 msgr2=0x7f8fe405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 shutdown_connections 2024-02-20T00:26:54.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:54.189+0000 7f8fe9422700 1 -- 172.21.15.195:0/2447431672 wait complete. 2024-02-20T00:26:54.192 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:26:54.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:52.532672+0000 mgr.smithi087.iydsqo (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:55.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:55 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:54.188431+0000 mon.smithi087 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.195:0/2447431672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:26:55.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:26:55.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:26:56.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:54.533042+0000 mgr.smithi087.iydsqo (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:56.664211+0000 mon.smithi087 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:26:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:56.666220+0000 mon.smithi087 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:26:57.548 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 -- 172.21.15.195:0/1273187116 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 msgr2=0x7f12e00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 --2- 172.21.15.195:0/1273187116 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e00f8f20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f12dc009fb0 tx=0x7f12dc009b30).stop 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 -- 172.21.15.195:0/1273187116 shutdown_connections 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 --2- 172.21.15.195:0/1273187116 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e00f8f20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 -- 172.21.15.195:0/1273187116 >> 172.21.15.195:0/1273187116 conn(0x7f12e00f09a0 msgr2=0x7f12e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 -- 172.21.15.195:0/1273187116 shutdown_connections 2024-02-20T00:26:57.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.542+0000 7f12e6c92700 1 -- 172.21.15.195:0/1273187116 wait complete. 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e6c92700 1 Processor -- start 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e6c92700 1 -- start start 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e6c92700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e6c92700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0106af0 con 0x7f12e00601e0 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e5c90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e5c90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47450/0 (socket says 172.21.15.195:47450) 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e5c90700 1 -- 172.21.15.195:0/476136938 learned_addr learned my addr 172.21.15.195:0/476136938 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.543+0000 7f12e5c90700 1 -- 172.21.15.195:0/476136938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc005040 con 0x7f12e00601e0 2024-02-20T00:26:57.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12e5c90700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f12dc005010 tx=0x7f12dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc009320 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0106cf0 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12dc013380 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc014e50 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.544+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e0107180 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0045ad0 con 0x7f12e00601e0 2024-02-20T00:26:57.551 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12dc0095f0 con 0x7f12e00601e0 2024-02-20T00:26:57.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12d67fc700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 0x7f12cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:26:57.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12dc040e90 con 0x7f12e00601e0 2024-02-20T00:26:57.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12e548f700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 0x7f12cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:26:57.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.545+0000 7f12e548f700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 0x7f12cc0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:26:57.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.549+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12dc014420 con 0x7f12e00601e0 2024-02-20T00:26:57.757 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.755+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12e0108030 con 0x7f12e00601e0 2024-02-20T00:26:57.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.756+0000 7f12d67fc700 1 -- 172.21.15.195:0/476136938 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12dc010230 con 0x7f12e00601e0 2024-02-20T00:26:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:26:57.758 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:26:57.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.757+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 msgr2=0x7f12cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.757+0000 7f12e6c92700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 0x7f12cc0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).stop 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.757+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 msgr2=0x7f12e0106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.757+0000 7f12e6c92700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f12dc005010 tx=0x7f12dc00b040).stop 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 shutdown_connections 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f12cc02ff10 0x7f12cc0323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 --2- 172.21.15.195:0/476136938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12e00601e0 0x7f12e0106600 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 >> 172.21.15.195:0/476136938 conn(0x7f12e00f09a0 msgr2=0x7f12e00f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 shutdown_connections 2024-02-20T00:26:57.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:26:57.758+0000 7f12e6c92700 1 -- 172.21.15.195:0/476136938 wait complete. 2024-02-20T00:26:57.761 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:26:58.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:56.533367+0000 mgr.smithi087.iydsqo (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:26:58.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:26:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:26:57.757276+0000 mon.smithi087 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.195:0/476136938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:00.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:26:58.533822+0000 mgr.smithi087.iydsqo (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:01.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:01.799 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:02.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:00.534135+0000 mgr.smithi087.iydsqo (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:02.534449+0000 mgr.smithi087.iydsqo (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:06.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:04.534834+0000 mgr.smithi087.iydsqo (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:07.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 -- 172.21.15.195:0/2308018786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 msgr2=0x7eff540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 --2- 172.21.15.195:0/2308018786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff540f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7eff44009fb0 tx=0x7eff44009b30).stop 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 -- 172.21.15.195:0/2308018786 shutdown_connections 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 --2- 172.21.15.195:0/2308018786 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff540f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 -- 172.21.15.195:0/2308018786 >> 172.21.15.195:0/2308018786 conn(0x7eff540f1c70 msgr2=0x7eff540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 -- 172.21.15.195:0/2308018786 shutdown_connections 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.218+0000 7eff5a316700 1 -- 172.21.15.195:0/2308018786 wait complete. 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff5a316700 1 Processor -- start 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff5a316700 1 -- start start 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff5a316700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:07.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff5a316700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff54106cd0 con 0x7eff540f6510 2024-02-20T00:27:07.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff53fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:07.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff53fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38158/0 (socket says 172.21.15.195:38158) 2024-02-20T00:27:07.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.219+0000 7eff53fff700 1 -- 172.21.15.195:0/1968187662 learned_addr learned my addr 172.21.15.195:0/1968187662 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:07.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff53fff700 1 -- 172.21.15.195:0/1968187662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44005040 con 0x7eff540f6510 2024-02-20T00:27:07.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff53fff700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7eff44005010 tx=0x7eff4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff44009320 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff54106ed0 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff44013380 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff44014e50 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.220+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff54107360 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.221+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff54045ad0 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.221+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff44014420 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.221+0000 7eff50ff9700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 0x7eff3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.221+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff44040e60 con 0x7eff540f6510 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.221+0000 7eff537fe700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 0x7eff3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:07.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.222+0000 7eff537fe700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 0x7eff3c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff480096a0 tx=0x7eff48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:07.227 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.226+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff440095f0 con 0x7eff540f6510 2024-02-20T00:27:07.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.431+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff541081b0 con 0x7eff540f6510 2024-02-20T00:27:07.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.432+0000 7eff50ff9700 1 -- 172.21.15.195:0/1968187662 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff4401db70 con 0x7eff540f6510 2024-02-20T00:27:07.434 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:07.435 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 msgr2=0x7eff3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 0x7eff3c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7eff480096a0 tx=0x7eff48006b40).stop 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 msgr2=0x7eff541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7eff44005010 tx=0x7eff4400b040).stop 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 shutdown_connections 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff3c02ff10 0x7eff3c0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 --2- 172.21.15.195:0/1968187662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff540f6510 0x7eff541067e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 >> 172.21.15.195:0/1968187662 conn(0x7eff540f1c70 msgr2=0x7eff540f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 shutdown_connections 2024-02-20T00:27:07.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:07.434+0000 7eff5a316700 1 -- 172.21.15.195:0/1968187662 wait complete. 2024-02-20T00:27:07.437 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:08.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:06.535064+0000 mgr.smithi087.iydsqo (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:08.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:07.433379+0000 mon.smithi087 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.195:0/1968187662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:09.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:09.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:10.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:08.535437+0000 mgr.smithi087.iydsqo (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:10.914 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 -- 172.21.15.195:0/1603748062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 msgr2=0x7f5fdc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:10.914 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/1603748062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc0f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5fcc009fb0 tx=0x7f5fcc009b30).stop 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 -- 172.21.15.195:0/1603748062 shutdown_connections 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/1603748062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc0f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 -- 172.21.15.195:0/1603748062 >> 172.21.15.195:0/1603748062 conn(0x7f5fdc0f1c70 msgr2=0x7f5fdc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 -- 172.21.15.195:0/1603748062 shutdown_connections 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.912+0000 7f5fe22bb700 1 -- 172.21.15.195:0/1603748062 wait complete. 2024-02-20T00:27:10.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fe22bb700 1 Processor -- start 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fe22bb700 1 -- start start 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fe22bb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fe22bb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc106cd0 con 0x7f5fdc0f6510 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fdbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fdbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38182/0 (socket says 172.21.15.195:38182) 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.913+0000 7f5fdbfff700 1 -- 172.21.15.195:0/3330998188 learned_addr learned my addr 172.21.15.195:0/3330998188 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fdbfff700 1 -- 172.21.15.195:0/3330998188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc005040 con 0x7f5fdc0f6510 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fdbfff700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5fcc0089f0 tx=0x7f5fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fcc009320 con 0x7f5fdc0f6510 2024-02-20T00:27:10.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc106ed0 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fcc013380 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fcc01ed80 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.914+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc107360 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.915+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fcc01e420 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.915+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc045ad0 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.915+0000 7f5fd8ff9700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 0x7f5fc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.915+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fcc0410f0 con 0x7f5fdc0f6510 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.915+0000 7f5fdb7fe700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 0x7f5fc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:10.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.916+0000 7f5fdb7fe700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 0x7f5fc4032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5fd00096a0 tx=0x7f5fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:10.921 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:10.920+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fcc01ccf0 con 0x7f5fdc0f6510 2024-02-20T00:27:11.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.126+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fdc108190 con 0x7f5fdc0f6510 2024-02-20T00:27:11.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.127+0000 7f5fd8ff9700 1 -- 172.21.15.195:0/3330998188 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fcc01cb10 con 0x7f5fdc0f6510 2024-02-20T00:27:11.128 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:11.129 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:11.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 msgr2=0x7f5fc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:11.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 0x7f5fc4032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5fd00096a0 tx=0x7f5fd0006b40).stop 2024-02-20T00:27:11.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 msgr2=0x7f5fdc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:11.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5fcc0089f0 tx=0x7f5fcc00b040).stop 2024-02-20T00:27:11.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 shutdown_connections 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f5fc402ff60 0x7f5fc4032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 --2- 172.21.15.195:0/3330998188 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5fdc0f6510 0x7f5fdc1067e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 >> 172.21.15.195:0/3330998188 conn(0x7f5fdc0f1c70 msgr2=0x7f5fdc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 shutdown_connections 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:11.128+0000 7f5fe22bb700 1 -- 172.21.15.195:0/3330998188 wait complete. 2024-02-20T00:27:11.131 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:12.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:10.535729+0000 mgr.smithi087.iydsqo (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:12.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:11.127940+0000 mon.smithi087 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.195:0/3330998188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:13.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:13.021 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:14.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:12.535966+0000 mgr.smithi087.iydsqo (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:14.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- 172.21.15.195:0/2771327142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 msgr2=0x7f66d00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2771327142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d00f8f20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f66c0009fb0 tx=0x7f66c0009b30).stop 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- 172.21.15.195:0/2771327142 shutdown_connections 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2771327142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d00f8f20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- 172.21.15.195:0/2771327142 >> 172.21.15.195:0/2771327142 conn(0x7f66d00f09a0 msgr2=0x7f66d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- 172.21.15.195:0/2771327142 shutdown_connections 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- 172.21.15.195:0/2771327142 wait complete. 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 Processor -- start 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.779+0000 7f66d4b15700 1 -- start start 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66d4b15700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66d4b15700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0106af0 con 0x7f66d00601e0 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38190/0 (socket says 172.21.15.195:38190) 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66cf7fe700 1 -- 172.21.15.195:0/2086251885 learned_addr learned my addr 172.21.15.195:0/2086251885 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:14.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66cf7fe700 1 -- 172.21.15.195:0/2086251885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0005040 con 0x7f66d00601e0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66cf7fe700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f66c0005010 tx=0x7f66c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.780+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c0009320 con 0x7f66d00601e0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.781+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d0106cf0 con 0x7f66d00601e0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.781+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66c0013380 con 0x7f66d00601e0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.781+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c0014e50 con 0x7f66d00601e0 2024-02-20T00:27:14.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.781+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d0107180 con 0x7f66d00601e0 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66c0014420 con 0x7f66d00601e0 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0045ad0 con 0x7f66d00601e0 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66b7fff700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 0x7f66b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66c0040e30 con 0x7f66d00601e0 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66ceffd700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 0x7f66b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:14.785 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.782+0000 7f66ceffd700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 0x7f66b80323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f66c40096a0 tx=0x7f66c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:14.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.786+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66c00095f0 con 0x7f66d00601e0 2024-02-20T00:27:14.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.993+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66d0108030 con 0x7f66d00601e0 2024-02-20T00:27:14.995 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.994+0000 7f66b7fff700 1 -- 172.21.15.195:0/2086251885 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66c001db70 con 0x7f66d00601e0 2024-02-20T00:27:14.996 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:14.996 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 msgr2=0x7f66b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 0x7f66b80323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f66c40096a0 tx=0x7f66c4006b40).stop 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 msgr2=0x7f66d0106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f66c0005010 tx=0x7f66c000b040).stop 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 shutdown_connections 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66b802ff10 0x7f66b80323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 --2- 172.21.15.195:0/2086251885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66d00601e0 0x7f66d0106600 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 >> 172.21.15.195:0/2086251885 conn(0x7f66d00f09a0 msgr2=0x7f66d00f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 shutdown_connections 2024-02-20T00:27:14.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:14.996+0000 7f66d4b15700 1 -- 172.21.15.195:0/2086251885 wait complete. 2024-02-20T00:27:14.999 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:15.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:15 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:14.994901+0000 mon.smithi087 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.195:0/2086251885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:14.536269+0000 mgr.smithi087.iydsqo (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:16.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:16.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:18.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.247+0000 7f47f358c700 1 -- 172.21.15.195:0/2681612190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 msgr2=0x7f47ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:18.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.247+0000 7f47f358c700 1 --2- 172.21.15.195:0/2681612190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec0f55c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f47dc009fb0 tx=0x7f47dc009b30).stop 2024-02-20T00:27:18.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- 172.21.15.195:0/2681612190 shutdown_connections 2024-02-20T00:27:18.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 --2- 172.21.15.195:0/2681612190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec0f55c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- 172.21.15.195:0/2681612190 >> 172.21.15.195:0/2681612190 conn(0x7f47ec0f09a0 msgr2=0x7f47ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- 172.21.15.195:0/2681612190 shutdown_connections 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- 172.21.15.195:0/2681612190 wait complete. 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 Processor -- start 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- start start 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f358c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec108d40 con 0x7f47ec0f51a0 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f258a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f258a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40174/0 (socket says 172.21.15.195:40174) 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.248+0000 7f47f258a700 1 -- 172.21.15.195:0/678321410 learned_addr learned my addr 172.21.15.195:0/678321410 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:18.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47f258a700 1 -- 172.21.15.195:0/678321410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc005040 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47f258a700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f47dc000c00 tx=0x7f47dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47dc0095f0 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec108f40 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47dc007e90 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47dc014db0 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.249+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec1093d0 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47dc025070 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec045ad0 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47e2ffd700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47dc040950 con 0x7f47ec0f51a0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47f1d89700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:18.254 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.250+0000 7f47f1d89700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 0x7f47d8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f47e80096a0 tx=0x7f47e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:18.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.255+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47dc00f300 con 0x7f47ec0f51a0 2024-02-20T00:27:18.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.460+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47ec10a220 con 0x7f47ec0f51a0 2024-02-20T00:27:18.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.461+0000 7f47e2ffd700 1 -- 172.21.15.195:0/678321410 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47dc00f300 con 0x7f47ec0f51a0 2024-02-20T00:27:18.463 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:18.463 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:18.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 msgr2=0x7f47d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 0x7f47d8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f47e80096a0 tx=0x7f47e8006b40).stop 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 msgr2=0x7f47ec108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f47dc000c00 tx=0x7f47dc00b040).stop 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 shutdown_connections 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 --2- 172.21.15.195:0/678321410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f47ec0f51a0 0x7f47ec108850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 >> 172.21.15.195:0/678321410 conn(0x7f47ec0f09a0 msgr2=0x7f47ec05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.462+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 shutdown_connections 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:18.463+0000 7f47f358c700 1 -- 172.21.15.195:0/678321410 wait complete. 2024-02-20T00:27:18.465 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:18.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:16.536544+0000 mgr.smithi087.iydsqo (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:19.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:19 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:18.461730+0000 mon.smithi087 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.195:0/678321410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:20.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:20.047 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:20.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:18.536916+0000 mgr.smithi087.iydsqo (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 -- 172.21.15.195:0/2324802473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 msgr2=0x7f112c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 --2- 172.21.15.195:0/2324802473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c0f66c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f111c009fb0 tx=0x7f111c009b30).stop 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 -- 172.21.15.195:0/2324802473 shutdown_connections 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 --2- 172.21.15.195:0/2324802473 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c0f66c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 -- 172.21.15.195:0/2324802473 >> 172.21.15.195:0/2324802473 conn(0x7f112c0f1aa0 msgr2=0x7f112c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.838+0000 7f1130b0a700 1 -- 172.21.15.195:0/2324802473 shutdown_connections 2024-02-20T00:27:21.841 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.839+0000 7f1130b0a700 1 -- 172.21.15.195:0/2324802473 wait complete. 2024-02-20T00:27:21.842 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.839+0000 7f1130b0a700 1 Processor -- start 2024-02-20T00:27:21.842 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.839+0000 7f1130b0a700 1 -- start start 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f1130b0a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f1130b0a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c106ac0 con 0x7f112c0f62a0 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40198/0 (socket says 172.21.15.195:40198) 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f112b7fe700 1 -- 172.21.15.195:0/203843046 learned_addr learned my addr 172.21.15.195:0/203843046 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.840+0000 7f112b7fe700 1 -- 172.21.15.195:0/203843046 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c005040 con 0x7f112c0f62a0 2024-02-20T00:27:21.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f112b7fe700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f111c0089f0 tx=0x7f111c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c009320 con 0x7f112c0f62a0 2024-02-20T00:27:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c106cc0 con 0x7f112c0f62a0 2024-02-20T00:27:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f111c013380 con 0x7f112c0f62a0 2024-02-20T00:27:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c014e20 con 0x7f112c0f62a0 2024-02-20T00:27:21.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.841+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c107150 con 0x7f112c0f62a0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.842+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f111c0095f0 con 0x7f112c0f62a0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.842+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112c045ad0 con 0x7f112c0f62a0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.842+0000 7f1113fff700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 0x7f11140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.842+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f111c040e90 con 0x7f112c0f62a0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.842+0000 7f112affd700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 0x7f11140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.843+0000 7f112affd700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 0x7f11140323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f11200096a0 tx=0x7f1120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:21.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:21.846+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f111c014420 con 0x7f112c0f62a0 2024-02-20T00:27:22.051 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.049+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f112c108000 con 0x7f112c0f62a0 2024-02-20T00:27:22.051 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.049+0000 7f1113fff700 1 -- 172.21.15.195:0/203843046 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f111c01d980 con 0x7f112c0f62a0 2024-02-20T00:27:22.051 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:22.051 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:22.056 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 msgr2=0x7f11140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:22.056 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 0x7f11140323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f11200096a0 tx=0x7f1120006b40).stop 2024-02-20T00:27:22.056 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 msgr2=0x7f112c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f111c0089f0 tx=0x7f111c00b040).stop 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 shutdown_connections 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f111402ff10 0x7f11140323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 --2- 172.21.15.195:0/203843046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f112c0f62a0 0x7f112c1065d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 >> 172.21.15.195:0/203843046 conn(0x7f112c0f1aa0 msgr2=0x7f112c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 shutdown_connections 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:22.051+0000 7f1130b0a700 1 -- 172.21.15.195:0/203843046 wait complete. 2024-02-20T00:27:22.057 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:20.537223+0000 mgr.smithi087.iydsqo (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:22.050627+0000 mon.smithi087 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.195:0/203843046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:23.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:23.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:24.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:22.537501+0000 mgr.smithi087.iydsqo (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 -- 172.21.15.195:0/1322059617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 msgr2=0x7fd3b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/1322059617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b00f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd3a4009fb0 tx=0x7fd3a4009b30).stop 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 -- 172.21.15.195:0/1322059617 shutdown_connections 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/1322059617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b00f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 -- 172.21.15.195:0/1322059617 >> 172.21.15.195:0/1322059617 conn(0x7fd3b00f0b70 msgr2=0x7fd3b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:25.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 -- 172.21.15.195:0/1322059617 shutdown_connections 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.346+0000 7fd3b6d88700 1 -- 172.21.15.195:0/1322059617 wait complete. 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.347+0000 7fd3b6d88700 1 Processor -- start 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.347+0000 7fd3b6d88700 1 -- start start 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.347+0000 7fd3b6d88700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.347+0000 7fd3b6d88700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b0105c10 con 0x7fd3b00f5410 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.347+0000 7fd3b4b24700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b4b24700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40210/0 (socket says 172.21.15.195:40210) 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b4b24700 1 -- 172.21.15.195:0/3430629312 learned_addr learned my addr 172.21.15.195:0/3430629312 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b4b24700 1 -- 172.21.15.195:0/3430629312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4005040 con 0x7fd3b00f5410 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b4b24700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd3a4005010 tx=0x7fd3a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a4009320 con 0x7fd3b00f5410 2024-02-20T00:27:25.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b0105e10 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a4013380 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a4014e50 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.348+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b01062a0 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.349+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3a40095f0 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.349+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b0045ad0 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.349+0000 7fd3ad7fa700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 0x7fd3a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.349+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3a4040f80 con 0x7fd3b00f5410 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.349+0000 7fd3affff700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 0x7fd3a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.350+0000 7fd3affff700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 0x7fd3a00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd39c0096a0 tx=0x7fd39c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:25.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.354+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3a4014420 con 0x7fd3b00f5410 2024-02-20T00:27:25.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.556+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3b0107150 con 0x7fd3b00f5410 2024-02-20T00:27:25.559 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.557+0000 7fd3ad7fa700 1 -- 172.21.15.195:0/3430629312 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3a4010230 con 0x7fd3b00f5410 2024-02-20T00:27:25.559 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:25.559 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.558+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 msgr2=0x7fd3a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.558+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 0x7fd3a00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd39c0096a0 tx=0x7fd39c006b40).stop 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.558+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 msgr2=0x7fd3b0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.558+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd3a4005010 tx=0x7fd3a400b040).stop 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 shutdown_connections 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3a002ff10 0x7fd3a00323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 --2- 172.21.15.195:0/3430629312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3b00f5410 0x7fd3b0105720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 >> 172.21.15.195:0/3430629312 conn(0x7fd3b00f0b70 msgr2=0x7fd3b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 shutdown_connections 2024-02-20T00:27:25.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:25.559+0000 7fd3b6d88700 1 -- 172.21.15.195:0/3430629312 wait complete. 2024-02-20T00:27:25.562 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:24.537818+0000 mgr.smithi087.iydsqo (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:26.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:25.558045+0000 mon.smithi087 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.195:0/3430629312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:27.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:27.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:28.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:26.538136+0000 mgr.smithi087.iydsqo (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:29.014 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- 172.21.15.195:0/3806120139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 msgr2=0x7fe7040f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:29.014 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 --2- 172.21.15.195:0/3806120139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe7040f8fc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe700009fb0 tx=0x7fe700009b30).stop 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- 172.21.15.195:0/3806120139 shutdown_connections 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 --2- 172.21.15.195:0/3806120139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe7040f8fc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- 172.21.15.195:0/3806120139 >> 172.21.15.195:0/3806120139 conn(0x7fe7040f09a0 msgr2=0x7fe7040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- 172.21.15.195:0/3806120139 shutdown_connections 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- 172.21.15.195:0/3806120139 wait complete. 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 Processor -- start 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.012+0000 7fe70b09b700 1 -- start start 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70b09b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:29.015 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70b09b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe704106af0 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70a099700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70a099700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44022/0 (socket says 172.21.15.195:44022) 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70a099700 1 -- 172.21.15.195:0/4074328545 learned_addr learned my addr 172.21.15.195:0/4074328545 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.013+0000 7fe70a099700 1 -- 172.21.15.195:0/4074328545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700005040 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe70a099700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe700005010 tx=0x7fe70000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe700009320 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe704106cf0 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe700013380 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe700014e50 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.014+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe704107180 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7000263a0 con 0x7fe704060140 2024-02-20T00:27:29.017 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe6faffd700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 0x7fe6f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:29.018 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe700040fc0 con 0x7fe704060140 2024-02-20T00:27:29.018 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe704045ad0 con 0x7fe704060140 2024-02-20T00:27:29.018 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe709898700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 0x7fe6f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:29.018 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.015+0000 7fe709898700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 0x7fe6f00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe6f40096a0 tx=0x7fe6f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:29.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.020+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe700012920 con 0x7fe704060140 2024-02-20T00:27:29.226 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.224+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe704107ea0 con 0x7fe704060140 2024-02-20T00:27:29.226 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.225+0000 7fe6faffd700 1 -- 172.21.15.195:0/4074328545 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe700012330 con 0x7fe704060140 2024-02-20T00:27:29.227 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:29.227 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:29.228 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 msgr2=0x7fe6f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:29.228 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 0x7fe6f00323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe6f40096a0 tx=0x7fe6f4006b40).stop 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 msgr2=0x7fe704106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe700005010 tx=0x7fe70000b040).stop 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 shutdown_connections 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6f002ff10 0x7fe6f00323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 --2- 172.21.15.195:0/4074328545 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe704060140 0x7fe704106600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 >> 172.21.15.195:0/4074328545 conn(0x7fe7040f09a0 msgr2=0x7fe7040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 shutdown_connections 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:29.226+0000 7fe70b09b700 1 -- 172.21.15.195:0/4074328545 wait complete. 2024-02-20T00:27:29.229 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:30.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:28.538444+0000 mgr.smithi087.iydsqo (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:30.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:29.225914+0000 mon.smithi087 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.195:0/4074328545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:30.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:30.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:32.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:30.538794+0000 mgr.smithi087.iydsqo (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:32.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 -- 172.21.15.195:0/4160118735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 msgr2=0x7f09840f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:32.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 --2- 172.21.15.195:0/4160118735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f09840f5df0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0978009fb0 tx=0x7f0978009b30).stop 2024-02-20T00:27:32.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 -- 172.21.15.195:0/4160118735 shutdown_connections 2024-02-20T00:27:32.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 --2- 172.21.15.195:0/4160118735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f09840f5df0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 -- 172.21.15.195:0/4160118735 >> 172.21.15.195:0/4160118735 conn(0x7f09840f02e0 msgr2=0x7f09840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 -- 172.21.15.195:0/4160118735 shutdown_connections 2024-02-20T00:27:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.678+0000 7f098937d700 1 -- 172.21.15.195:0/4160118735 wait complete. 2024-02-20T00:27:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f098937d700 1 Processor -- start 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f098937d700 1 -- start start 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f098937d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f098937d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09841072d0 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f0982ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f0982ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44044/0 (socket says 172.21.15.195:44044) 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f0982ffd700 1 -- 172.21.15.195:0/1334637780 learned_addr learned my addr 172.21.15.195:0/1334637780 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.679+0000 7f0982ffd700 1 -- 172.21.15.195:0/1334637780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978005040 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f0982ffd700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0978005010 tx=0x7f097800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0978009320 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09840f5390 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098410bdd0 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0978013380 con 0x7f09840f59d0 2024-02-20T00:27:32.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.680+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0978014df0 con 0x7f09840f59d0 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09780095f0 con 0x7f09840f59d0 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0984045ad0 con 0x7f09840f59d0 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f096b7fe700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 0x7f096c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0978040eb0 con 0x7f09840f59d0 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f09827fc700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 0x7f096c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:32.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.681+0000 7f09827fc700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 0x7f096c0323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f09740096a0 tx=0x7f0974006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:32.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.685+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f097801dbb0 con 0x7f09840f59d0 2024-02-20T00:27:32.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.892+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f098410c590 con 0x7f09840f59d0 2024-02-20T00:27:32.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.892+0000 7f096b7fe700 1 -- 172.21.15.195:0/1334637780 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f097801d9d0 con 0x7f09840f59d0 2024-02-20T00:27:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:32.894 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 msgr2=0x7f096c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 0x7f096c0323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f09740096a0 tx=0x7f0974006b40).stop 2024-02-20T00:27:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 msgr2=0x7f098410b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0978005010 tx=0x7f097800b040).stop 2024-02-20T00:27:32.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 shutdown_connections 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f096c02ff10 0x7f096c0323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 --2- 172.21.15.195:0/1334637780 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09840f59d0 0x7f098410b8e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 >> 172.21.15.195:0/1334637780 conn(0x7f09840f02e0 msgr2=0x7f09840f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 shutdown_connections 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:32.894+0000 7f098937d700 1 -- 172.21.15.195:0/1334637780 wait complete. 2024-02-20T00:27:32.897 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:33.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:32.893614+0000 mon.smithi087 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.195:0/1334637780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:34.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:34.525 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:32.539085+0000 mgr.smithi087.iydsqo (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 -- 172.21.15.195:0/3355806126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 msgr2=0x7f48280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 --2- 172.21.15.195:0/3355806126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48280f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f481c009fb0 tx=0x7f481c009b30).stop 2024-02-20T00:27:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 -- 172.21.15.195:0/3355806126 shutdown_connections 2024-02-20T00:27:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 --2- 172.21.15.195:0/3355806126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48280f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:36.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 -- 172.21.15.195:0/3355806126 >> 172.21.15.195:0/3355806126 conn(0x7f48280f0b70 msgr2=0x7f48280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 -- 172.21.15.195:0/3355806126 shutdown_connections 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.299+0000 7f482f4e1700 1 -- 172.21.15.195:0/3355806126 wait complete. 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482f4e1700 1 Processor -- start 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482f4e1700 1 -- start start 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482f4e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482f4e1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828105bd0 con 0x7f48280f5410 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482d27d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482d27d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44052/0 (socket says 172.21.15.195:44052) 2024-02-20T00:27:36.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.300+0000 7f482d27d700 1 -- 172.21.15.195:0/1622798062 learned_addr learned my addr 172.21.15.195:0/1622798062 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f482d27d700 1 -- 172.21.15.195:0/1622798062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c005040 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f482d27d700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f481c005010 tx=0x7f481c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f481c009320 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4828105dd0 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f481c013380 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f481c014e50 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.301+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4828106260 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.302+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f481c014420 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.302+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4828045ad0 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.302+0000 7f4819ffb700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 0x7f48140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.302+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f481c040e60 con 0x7f48280f5410 2024-02-20T00:27:36.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.302+0000 7f482ca7c700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 0x7f48140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:36.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.303+0000 7f482ca7c700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 0x7f48140323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:36.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.307+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f481c0095f0 con 0x7f48280f5410 2024-02-20T00:27:36.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.511+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4828107110 con 0x7f48280f5410 2024-02-20T00:27:36.513 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.511+0000 7f4819ffb700 1 -- 172.21.15.195:0/1622798062 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f481c01db70 con 0x7f48280f5410 2024-02-20T00:27:36.513 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:36.513 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 msgr2=0x7f48140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 0x7f48140323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).stop 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 msgr2=0x7f48281056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f481c005010 tx=0x7f481c00b040).stop 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 shutdown_connections 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f481402ff10 0x7f48140323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 --2- 172.21.15.195:0/1622798062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48280f5410 0x7f48281056e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 >> 172.21.15.195:0/1622798062 conn(0x7f48280f0b70 msgr2=0x7f48280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 shutdown_connections 2024-02-20T00:27:36.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:36.513+0000 7f482f4e1700 1 -- 172.21.15.195:0/1622798062 wait complete. 2024-02-20T00:27:36.516 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:36.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:34.539400+0000 mgr.smithi087.iydsqo (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:37.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:36.512502+0000 mon.smithi087 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.195:0/1622798062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:38.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:36.539821+0000 mgr.smithi087.iydsqo (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:38.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:38.719 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 -- 172.21.15.195:0/43503533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 msgr2=0x7ff318069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 --2- 172.21.15.195:0/43503533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff318069e60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff30c009fb0 tx=0x7ff30c009b30).stop 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 -- 172.21.15.195:0/43503533 shutdown_connections 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 --2- 172.21.15.195:0/43503533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff318069e60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 -- 172.21.15.195:0/43503533 >> 172.21.15.195:0/43503533 conn(0x7ff3180f1c70 msgr2=0x7ff3180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 -- 172.21.15.195:0/43503533 shutdown_connections 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.509+0000 7ff31ddda700 1 -- 172.21.15.195:0/43503533 wait complete. 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff31ddda700 1 Processor -- start 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff31ddda700 1 -- start start 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff31ddda700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:40.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff31ddda700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff318108e90 con 0x7ff31806b960 2024-02-20T00:27:40.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff3177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:40.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff3177fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:54172/0 (socket says 172.21.15.195:54172) 2024-02-20T00:27:40.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff3177fe700 1 -- 172.21.15.195:0/1198030917 learned_addr learned my addr 172.21.15.195:0/1198030917 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:40.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.510+0000 7ff3177fe700 1 -- 172.21.15.195:0/1198030917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c005040 con 0x7ff31806b960 2024-02-20T00:27:40.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff3177fe700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff30c0089f0 tx=0x7ff30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c009320 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff318109090 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff30c013380 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30c014e20 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.511+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff318109520 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff30c014420 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff318045ad0 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff2fffff700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 0x7ff3000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff30c040f50 con 0x7ff31806b960 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff316ffd700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 0x7ff3000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:40.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.512+0000 7ff316ffd700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 0x7ff3000323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff3080096a0 tx=0x7ff308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:40.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.517+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff30c06d2b0 con 0x7ff31806b960 2024-02-20T00:27:40.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:38.540162+0000 mgr.smithi087.iydsqo (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:40.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.722+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff31810a3d0 con 0x7ff31806b960 2024-02-20T00:27:40.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.723+0000 7ff2fffff700 1 -- 172.21.15.195:0/1198030917 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff30c0095f0 con 0x7ff31806b960 2024-02-20T00:27:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:40.724 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:40.725 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 msgr2=0x7ff3000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 0x7ff3000323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff3080096a0 tx=0x7ff308006b40).stop 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 msgr2=0x7ff3181089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff30c0089f0 tx=0x7ff30c00b040).stop 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 shutdown_connections 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff30002ff10 0x7ff3000323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 --2- 172.21.15.195:0/1198030917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff31806b960 0x7ff3181089a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 >> 172.21.15.195:0/1198030917 conn(0x7ff3180f1c70 msgr2=0x7ff3180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 shutdown_connections 2024-02-20T00:27:40.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:40.725+0000 7ff31ddda700 1 -- 172.21.15.195:0/1198030917 wait complete. 2024-02-20T00:27:40.727 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:41.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:40.723328+0000 mon.smithi087 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.195:0/1198030917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:42.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:42.295 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:42.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:40.540503+0000 mgr.smithi087.iydsqo (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 -- 172.21.15.195:0/833634114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 msgr2=0x7fd9d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 --2- 172.21.15.195:0/833634114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d40f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fd9c8009fb0 tx=0x7fd9c8009b30).stop 2024-02-20T00:27:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 -- 172.21.15.195:0/833634114 shutdown_connections 2024-02-20T00:27:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 --2- 172.21.15.195:0/833634114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d40f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 -- 172.21.15.195:0/833634114 >> 172.21.15.195:0/833634114 conn(0x7fd9d40f0b70 msgr2=0x7fd9d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 -- 172.21.15.195:0/833634114 shutdown_connections 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.335+0000 7fd9dc789700 1 -- 172.21.15.195:0/833634114 wait complete. 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9dc789700 1 Processor -- start 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9dc789700 1 -- start start 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9dc789700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9dc789700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41072f0 con 0x7fd9d40f5410 2024-02-20T00:27:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9da525700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9da525700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:54202/0 (socket says 172.21.15.195:54202) 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.336+0000 7fd9da525700 1 -- 172.21.15.195:0/3090105274 learned_addr learned my addr 172.21.15.195:0/3090105274 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9da525700 1 -- 172.21.15.195:0/3090105274 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c8005040 con 0x7fd9d40f5410 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9da525700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9c8005010 tx=0x7fd9c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c8009320 con 0x7fd9d40f5410 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d4060220 con 0x7fd9d40f5410 2024-02-20T00:27:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9c8013380 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c8014e50 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.337+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d410bdf0 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9c80095f0 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d4045ad0 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9c6ffd700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9c8040f80 con 0x7fd9d40f5410 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9d9d24700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:44.343 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.338+0000 7fd9d9d24700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd9d00096a0 tx=0x7fd9d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:44.344 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.342+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9c8014420 con 0x7fd9d40f5410 2024-02-20T00:27:44.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:42.540864+0000 mgr.smithi087.iydsqo (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:44.549 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.549+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9d410c5b0 con 0x7fd9d40f5410 2024-02-20T00:27:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.549+0000 7fd9c6ffd700 1 -- 172.21.15.195:0/3090105274 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9c8010230 con 0x7fd9d40f5410 2024-02-20T00:27:44.550 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:44.550 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:44.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 msgr2=0x7fd9c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:44.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd9d00096a0 tx=0x7fd9d0006b40).stop 2024-02-20T00:27:44.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 msgr2=0x7fd9d410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:44.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9c8005010 tx=0x7fd9c800b040).stop 2024-02-20T00:27:44.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 shutdown_connections 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 --2- 172.21.15.195:0/3090105274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d40f5410 0x7fd9d410b900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 >> 172.21.15.195:0/3090105274 conn(0x7fd9d40f0b70 msgr2=0x7fd9d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 shutdown_connections 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:44.551+0000 7fd9dc789700 1 -- 172.21.15.195:0/3090105274 wait complete. 2024-02-20T00:27:44.553 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:45.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:44.549565+0000 mon.smithi087 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.195:0/3090105274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:46.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:46.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:44.541182+0000 mgr.smithi087.iydsqo (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:48.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/521125650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 msgr2=0x7f7ef40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/521125650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef40f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f7ee4009fb0 tx=0x7f7ee4009b30).stop 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/521125650 shutdown_connections 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/521125650 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef40f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/521125650 >> 172.21.15.195:0/521125650 conn(0x7f7ef40f0b70 msgr2=0x7f7ef40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/521125650 shutdown_connections 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/521125650 wait complete. 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.142+0000 7f7ef9bc2700 1 Processor -- start 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef9bc2700 1 -- start start 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef9bc2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef9bc2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef4106ca0 con 0x7f7ef40f5410 2024-02-20T00:27:48.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:37204/0 (socket says 172.21.15.195:37204) 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef37fe700 1 -- 172.21.15.195:0/4228710813 learned_addr learned my addr 172.21.15.195:0/4228710813 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef37fe700 1 -- 172.21.15.195:0/4228710813 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4005040 con 0x7f7ef40f5410 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.143+0000 7f7ef37fe700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7ee40089f0 tx=0x7f7ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee4009320 con 0x7f7ef40f5410 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ee4013380 con 0x7f7ef40f5410 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef4106ea0 con 0x7f7ef40f5410 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee401ed80 con 0x7f7ef40f5410 2024-02-20T00:27:48.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef4107330 con 0x7f7ef40f5410 2024-02-20T00:27:48.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.144+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ee401e420 con 0x7f7ef40f5410 2024-02-20T00:27:48.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.145+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef4045ad0 con 0x7f7ef40f5410 2024-02-20T00:27:48.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.145+0000 7f7edbfff700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 0x7f7edc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:48.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.145+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ee4040f80 con 0x7f7ef40f5410 2024-02-20T00:27:48.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.145+0000 7f7ef2ffd700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 0x7f7edc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:48.149 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.145+0000 7f7ef2ffd700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 0x7f7edc032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7ee80096a0 tx=0x7f7ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:48.151 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.149+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ee406d2b0 con 0x7f7ef40f5410 2024-02-20T00:27:48.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.355+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ef41081e0 con 0x7f7ef40f5410 2024-02-20T00:27:48.357 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.355+0000 7f7edbfff700 1 -- 172.21.15.195:0/4228710813 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ee40095f0 con 0x7f7ef40f5410 2024-02-20T00:27:48.357 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:48.357 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 msgr2=0x7f7edc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 0x7f7edc032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7ee80096a0 tx=0x7f7ee8006b40).stop 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 msgr2=0x7f7ef41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7ee40089f0 tx=0x7f7ee400b040).stop 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 shutdown_connections 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7edc02ff60 0x7f7edc032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 --2- 172.21.15.195:0/4228710813 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7ef40f5410 0x7f7ef41067b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 >> 172.21.15.195:0/4228710813 conn(0x7f7ef40f0b70 msgr2=0x7f7ef405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 shutdown_connections 2024-02-20T00:27:48.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:48.357+0000 7f7ef9bc2700 1 -- 172.21.15.195:0/4228710813 wait complete. 2024-02-20T00:27:48.360 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:48.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:46.541445+0000 mgr.smithi087.iydsqo (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:49.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:48.356587+0000 mon.smithi087 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.195:0/4228710813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:49.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:49.933 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:50.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:48.541803+0000 mgr.smithi087.iydsqo (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 -- 172.21.15.195:0/880645326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 msgr2=0x7f10540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 --2- 172.21.15.195:0/880645326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10540f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1044009fb0 tx=0x7f1044009b30).stop 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 -- 172.21.15.195:0/880645326 shutdown_connections 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 --2- 172.21.15.195:0/880645326 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10540f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 -- 172.21.15.195:0/880645326 >> 172.21.15.195:0/880645326 conn(0x7f10540f1c70 msgr2=0x7f10540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:51.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 -- 172.21.15.195:0/880645326 shutdown_connections 2024-02-20T00:27:51.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 -- 172.21.15.195:0/880645326 wait complete. 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.615+0000 7f105b470700 1 Processor -- start 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105b470700 1 -- start start 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105b470700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105b470700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1054106cd0 con 0x7f10540f6510 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105920c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105920c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:37222/0 (socket says 172.21.15.195:37222) 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105920c700 1 -- 172.21.15.195:0/2201093669 learned_addr learned my addr 172.21.15.195:0/2201093669 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:51.619 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105920c700 1 -- 172.21.15.195:0/2201093669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044005040 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105920c700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1044005010 tx=0x7f104400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1044009320 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.616+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1054106ed0 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.617+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1044013380 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.617+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1044014e50 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.617+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1054107360 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.617+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1054045ad0 con 0x7f10540f6510 2024-02-20T00:27:51.620 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.618+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10440095f0 con 0x7f10540f6510 2024-02-20T00:27:51.621 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.618+0000 7f1049ffb700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 0x7f10400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:51.621 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.618+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1044040e90 con 0x7f10540f6510 2024-02-20T00:27:51.621 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.618+0000 7f1058a0b700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 0x7f10400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:51.621 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.618+0000 7f1058a0b700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 0x7f10400323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f10500096a0 tx=0x7f1050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:51.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.622+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1044014420 con 0x7f10540f6510 2024-02-20T00:27:51.828 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.826+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1054108210 con 0x7f10540f6510 2024-02-20T00:27:51.829 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.827+0000 7f1049ffb700 1 -- 172.21.15.195:0/2201093669 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1044010230 con 0x7f10540f6510 2024-02-20T00:27:51.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:51.830 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 msgr2=0x7f10400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 0x7f10400323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f10500096a0 tx=0x7f1050006b40).stop 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 msgr2=0x7f10541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1044005010 tx=0x7f104400b040).stop 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 shutdown_connections 2024-02-20T00:27:51.831 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f104002ff10 0x7f10400323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:51.832 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 --2- 172.21.15.195:0/2201093669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f10540f6510 0x7f10541067e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:51.832 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 >> 172.21.15.195:0/2201093669 conn(0x7f10540f1c70 msgr2=0x7f10540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:51.832 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 shutdown_connections 2024-02-20T00:27:51.832 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:51.829+0000 7f105b470700 1 -- 172.21.15.195:0/2201093669 wait complete. 2024-02-20T00:27:51.832 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:52.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:50.542090+0000 mgr.smithi087.iydsqo (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:52.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:51.828461+0000 mon.smithi087 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.195:0/2201093669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:53.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:53.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:52.542337+0000 mgr.smithi087.iydsqo (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:55.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- 172.21.15.195:0/205015520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 msgr2=0x7ff85c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 --2- 172.21.15.195:0/205015520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff84c009fb0 tx=0x7ff84c009b30).stop 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- 172.21.15.195:0/205015520 shutdown_connections 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 --2- 172.21.15.195:0/205015520 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- 172.21.15.195:0/205015520 >> 172.21.15.195:0/205015520 conn(0x7ff85c0f0b70 msgr2=0x7ff85c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- 172.21.15.195:0/205015520 shutdown_connections 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- 172.21.15.195:0/205015520 wait complete. 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 Processor -- start 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- start start 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.326+0000 7ff860b6a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0fb960 con 0x7ff85c0f5410 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff85a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff85a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:37248/0 (socket says 172.21.15.195:37248) 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff85a59c700 1 -- 172.21.15.195:0/620573289 learned_addr learned my addr 172.21.15.195:0/620573289 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:55.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff85a59c700 1 -- 172.21.15.195:0/620573289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c005040 con 0x7ff85c0f5410 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff85a59c700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff84c005010 tx=0x7ff84c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff84c009320 con 0x7ff85c0f5410 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c0fbb00 con 0x7ff85c0f5410 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff84c013380 con 0x7ff85c0f5410 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff84c014e50 con 0x7ff85c0f5410 2024-02-20T00:27:55.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.327+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c0fbf90 con 0x7ff85c0f5410 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.328+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff84c0095f0 con 0x7ff85c0f5410 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.328+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c059100 con 0x7ff85c0f5410 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.328+0000 7ff84affd700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 0x7ff8440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.328+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff84c040fa0 con 0x7ff85c0f5410 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.328+0000 7ff859d9b700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 0x7ff8440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:55.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.329+0000 7ff859d9b700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 0x7ff8440323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:55.335 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.333+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff84c014420 con 0x7ff85c0f5410 2024-02-20T00:27:55.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.537+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff85c0fafa0 con 0x7ff85c0f5410 2024-02-20T00:27:55.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.538+0000 7ff84affd700 1 -- 172.21.15.195:0/620573289 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff84c010230 con 0x7ff85c0f5410 2024-02-20T00:27:55.540 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:55.540 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:55.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 msgr2=0x7ff8440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:55.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 0x7ff8440323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).stop 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 msgr2=0x7ff85c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff84c005010 tx=0x7ff84c00b040).stop 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 shutdown_connections 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff84402ff10 0x7ff8440323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 --2- 172.21.15.195:0/620573289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff85c0f5410 0x7ff85c0fd4e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.539+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 >> 172.21.15.195:0/620573289 conn(0x7ff85c0f0b70 msgr2=0x7ff85c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.540+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 shutdown_connections 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:55.540+0000 7ff860b6a700 1 -- 172.21.15.195:0/620573289 wait complete. 2024-02-20T00:27:55.542 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:54.542678+0000 mgr.smithi087.iydsqo (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:55.539061+0000 mon.smithi087 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.195:0/620573289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:27:57.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:27:57.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:27:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:56.665268+0000 mon.smithi087 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:27:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:56.667128+0000 mon.smithi087 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:27:58.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:56.543027+0000 mgr.smithi087.iydsqo (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:27:58.888 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- 172.21.15.195:0/356908723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 msgr2=0x7fae8c0f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 --2- 172.21.15.195:0/356908723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c0f5810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fae80009fb0 tx=0x7fae80009b30).stop 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- 172.21.15.195:0/356908723 shutdown_connections 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 --2- 172.21.15.195:0/356908723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c0f5810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- 172.21.15.195:0/356908723 >> 172.21.15.195:0/356908723 conn(0x7fae8c0f0b70 msgr2=0x7fae8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- 172.21.15.195:0/356908723 shutdown_connections 2024-02-20T00:27:58.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- 172.21.15.195:0/356908723 wait complete. 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 Processor -- start 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 -- start start 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.887+0000 7fae931d5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae931d5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c105bb0 con 0x7fae8c0f53f0 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae90f71700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae90f71700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59070/0 (socket says 172.21.15.195:59070) 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae90f71700 1 -- 172.21.15.195:0/3895027845 learned_addr learned my addr 172.21.15.195:0/3895027845 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae90f71700 1 -- 172.21.15.195:0/3895027845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80005040 con 0x7fae8c0f53f0 2024-02-20T00:27:58.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae90f71700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fae80005010 tx=0x7fae8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:58.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae80009320 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.888+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c105db0 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae80013380 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae80014e50 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c106240 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae800263a0 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c045ad0 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae897fa700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 0x7fae7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.889+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae80041140 con 0x7fae8c0f53f0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.890+0000 7fae8bfff700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 0x7fae7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:27:58.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.890+0000 7fae8bfff700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 0x7fae7c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fae780096a0 tx=0x7fae78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:27:58.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:58.894+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae8001cb10 con 0x7fae8c0f53f0 2024-02-20T00:27:59.101 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.100+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae8c1070f0 con 0x7fae8c0f53f0 2024-02-20T00:27:59.102 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.100+0000 7fae897fa700 1 -- 172.21.15.195:0/3895027845 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae8001cb10 con 0x7fae8c0f53f0 2024-02-20T00:27:59.102 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:27:59.102 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:27:59.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 msgr2=0x7fae7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 0x7fae7c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fae780096a0 tx=0x7fae78006b40).stop 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 msgr2=0x7fae8c1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fae80005010 tx=0x7fae8000b040).stop 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 shutdown_connections 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fae7c02ff10 0x7fae7c0323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 --2- 172.21.15.195:0/3895027845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fae8c0f53f0 0x7fae8c1056c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 >> 172.21.15.195:0/3895027845 conn(0x7fae8c0f0b70 msgr2=0x7fae8c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 shutdown_connections 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:27:59.102+0000 7fae931d5700 1 -- 172.21.15.195:0/3895027845 wait complete. 2024-02-20T00:27:59.104 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:27:59.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:27:59 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:27:59.101524+0000 mon.smithi087 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.195:0/3895027845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:00.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:27:58.543379+0000 mgr.smithi087.iydsqo (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:00.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:00.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:02.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:00.543718+0000 mgr.smithi087.iydsqo (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:02.596 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.586+0000 7f0843f9d700 1 -- 172.21.15.195:0/3747396985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 msgr2=0x7f083c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.586+0000 7f0843f9d700 1 --2- 172.21.15.195:0/3747396985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0830009fb0 tx=0x7f0830009b30).stop 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.586+0000 7f0843f9d700 1 -- 172.21.15.195:0/3747396985 shutdown_connections 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.586+0000 7f0843f9d700 1 --2- 172.21.15.195:0/3747396985 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.586+0000 7f0843f9d700 1 -- 172.21.15.195:0/3747396985 >> 172.21.15.195:0/3747396985 conn(0x7f083c0f0b70 msgr2=0x7f083c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 -- 172.21.15.195:0/3747396985 shutdown_connections 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 -- 172.21.15.195:0/3747396985 wait complete. 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 Processor -- start 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 -- start start 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.587+0000 7f0843f9d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c060240 con 0x7f083c0f5410 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.588+0000 7f0841d39700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.588+0000 7f0841d39700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59088/0 (socket says 172.21.15.195:59088) 2024-02-20T00:28:02.597 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.588+0000 7f0841d39700 1 -- 172.21.15.195:0/330574929 learned_addr learned my addr 172.21.15.195:0/330574929 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.588+0000 7f0841d39700 1 -- 172.21.15.195:0/330574929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830005040 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.588+0000 7f0841d39700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0830005010 tx=0x7f083000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.589+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0830009320 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.589+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c060440 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.589+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0830013380 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.589+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0830014e50 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.589+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c109b90 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0830014420 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c045ad0 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f082e7fc700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 0x7f08280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0830040e30 con 0x7f083c0f5410 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f0841538700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 0x7f08280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.590+0000 7f0841538700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 0x7f08280323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f08380096a0 tx=0x7f0838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:02.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.594+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08300095f0 con 0x7f083c0f5410 2024-02-20T00:28:02.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.800+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f083c10a3b0 con 0x7f083c0f5410 2024-02-20T00:28:02.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.801+0000 7f082e7fc700 1 -- 172.21.15.195:0/330574929 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f083001db70 con 0x7f083c0f5410 2024-02-20T00:28:02.803 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:02.803 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 msgr2=0x7f08280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 0x7f08280323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f08380096a0 tx=0x7f0838006b40).stop 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 msgr2=0x7f083c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f0830005010 tx=0x7f083000b040).stop 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 shutdown_connections 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f082802ff10 0x7f08280323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 --2- 172.21.15.195:0/330574929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f083c0f5410 0x7f083c1094e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 >> 172.21.15.195:0/330574929 conn(0x7f083c0f0b70 msgr2=0x7f083c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 shutdown_connections 2024-02-20T00:28:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:02.803+0000 7f0843f9d700 1 -- 172.21.15.195:0/330574929 wait complete. 2024-02-20T00:28:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:03.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:03 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:02.802153+0000 mon.smithi087 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.195:0/330574929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:04.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:04.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:02.544015+0000 mgr.smithi087.iydsqo (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:06.426 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- 172.21.15.195:0/1937428262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 msgr2=0x7f3a380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:06.426 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 --2- 172.21.15.195:0/1937428262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a380f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3a28009fb0 tx=0x7f3a28009b30).stop 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- 172.21.15.195:0/1937428262 shutdown_connections 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 --2- 172.21.15.195:0/1937428262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a380f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- 172.21.15.195:0/1937428262 >> 172.21.15.195:0/1937428262 conn(0x7f3a380f09a0 msgr2=0x7f3a380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- 172.21.15.195:0/1937428262 shutdown_connections 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- 172.21.15.195:0/1937428262 wait complete. 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 Processor -- start 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.424+0000 7f3a3759e700 1 -- start start 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:06.427 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3759e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a381059c0 con 0x7f3a380f51a0 2024-02-20T00:28:06.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36516/0 (socket says 172.21.15.195:36516) 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3659c700 1 -- 172.21.15.195:0/3494090347 learned_addr learned my addr 172.21.15.195:0/3494090347 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3659c700 1 -- 172.21.15.195:0/3494090347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a28005040 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.425+0000 7f3a3659c700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3a280089f0 tx=0x7f3a2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.426+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a28009320 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.426+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38105bc0 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.426+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a38106050 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.426+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a28013380 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.426+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a28014e40 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a28014420 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38045ad0 con 0x7f3a380f51a0 2024-02-20T00:28:06.432 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a26ffd700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:06.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a28042d00 con 0x7f3a380f51a0 2024-02-20T00:28:06.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a35d9b700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:06.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.427+0000 7f3a35d9b700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 0x7f3a200323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3a2c0096a0 tx=0x7f3a2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:06.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.432+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a28046030 con 0x7f3a380f51a0 2024-02-20T00:28:06.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:04.544327+0000 mgr.smithi087.iydsqo (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:06.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.636+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a38106f00 con 0x7f3a380f51a0 2024-02-20T00:28:06.638 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.636+0000 7f3a26ffd700 1 -- 172.21.15.195:0/3494090347 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a28009680 con 0x7f3a380f51a0 2024-02-20T00:28:06.638 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:06.638 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 msgr2=0x7f3a200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 0x7f3a200323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3a2c0096a0 tx=0x7f3a2c006b40).stop 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 msgr2=0x7f3a381054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f3a280089f0 tx=0x7f3a2800b040).stop 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 shutdown_connections 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 --2- 172.21.15.195:0/3494090347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a380f51a0 0x7f3a381054d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 >> 172.21.15.195:0/3494090347 conn(0x7f3a380f09a0 msgr2=0x7f3a380f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 shutdown_connections 2024-02-20T00:28:06.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:06.638+0000 7f3a3759e700 1 -- 172.21.15.195:0/3494090347 wait complete. 2024-02-20T00:28:06.641 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:07.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:06.637728+0000 mon.smithi087 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.195:0/3494090347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:08.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:08.292 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:08.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:06.544696+0000 mgr.smithi087.iydsqo (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.148+0000 7f7d144bf700 1 -- 172.21.15.195:0/3024947603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 msgr2=0x7f7d0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.148+0000 7f7d144bf700 1 --2- 172.21.15.195:0/3024947603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c0f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f7cfc009fb0 tx=0x7f7cfc009b30).stop 2024-02-20T00:28:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.148+0000 7f7d144bf700 1 -- 172.21.15.195:0/3024947603 shutdown_connections 2024-02-20T00:28:10.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.148+0000 7f7d144bf700 1 --2- 172.21.15.195:0/3024947603 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c0f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.148+0000 7f7d144bf700 1 -- 172.21.15.195:0/3024947603 >> 172.21.15.195:0/3024947603 conn(0x7f7d0c0f1c70 msgr2=0x7f7d0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 -- 172.21.15.195:0/3024947603 shutdown_connections 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 -- 172.21.15.195:0/3024947603 wait complete. 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 Processor -- start 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 -- start start 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d144bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c106cd0 con 0x7f7d0c0f6510 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d1225b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d1225b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36534/0 (socket says 172.21.15.195:36534) 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.149+0000 7f7d1225b700 1 -- 172.21.15.195:0/1881732376 learned_addr learned my addr 172.21.15.195:0/1881732376 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d1225b700 1 -- 172.21.15.195:0/1881732376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc005040 con 0x7f7d0c0f6510 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d1225b700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7cfc0089f0 tx=0x7f7cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cfc009320 con 0x7f7d0c0f6510 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c106ed0 con 0x7f7d0c0f6510 2024-02-20T00:28:10.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cfc013380 con 0x7f7d0c0f6510 2024-02-20T00:28:10.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cfc01ed80 con 0x7f7d0c0f6510 2024-02-20T00:28:10.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.150+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c107360 con 0x7f7d0c0f6510 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cfc01e420 con 0x7f7d0c0f6510 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c045ad0 con 0x7f7d0c0f6510 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d02ffd700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 0x7f7cf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cfc041150 con 0x7f7d0c0f6510 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d11a5a700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 0x7f7cf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:10.155 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.151+0000 7f7d11a5a700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 0x7f7cf8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7d080096a0 tx=0x7f7d08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:10.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.156+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cfc0095f0 con 0x7f7d0c0f6510 2024-02-20T00:28:10.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.360+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d0c108210 con 0x7f7d0c0f6510 2024-02-20T00:28:10.362 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.361+0000 7f7d02ffd700 1 -- 172.21.15.195:0/1881732376 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cfc004490 con 0x7f7d0c0f6510 2024-02-20T00:28:10.363 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:10.363 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:10.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 msgr2=0x7f7cf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 0x7f7cf8032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7d080096a0 tx=0x7f7d08006b40).stop 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 msgr2=0x7f7d0c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7cfc0089f0 tx=0x7f7cfc00b040).stop 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 shutdown_connections 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7cf802ff60 0x7f7cf8032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.362+0000 7f7d144bf700 1 --2- 172.21.15.195:0/1881732376 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d0c0f6510 0x7f7d0c1067e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.363+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 >> 172.21.15.195:0/1881732376 conn(0x7f7d0c0f1c70 msgr2=0x7f7d0c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.363+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 shutdown_connections 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:10.363+0000 7f7d144bf700 1 -- 172.21.15.195:0/1881732376 wait complete. 2024-02-20T00:28:10.365 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:10.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:08.545004+0000 mgr.smithi087.iydsqo (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:11.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:10.362075+0000 mon.smithi087 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.195:0/1881732376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:12.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:12.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:12.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:10.545343+0000 mgr.smithi087.iydsqo (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:13.873 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 -- 172.21.15.195:0/3063312393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 msgr2=0x7f39cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/3063312393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc0f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f39bc009fb0 tx=0x7f39bc009b30).stop 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 -- 172.21.15.195:0/3063312393 shutdown_connections 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/3063312393 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc0f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 -- 172.21.15.195:0/3063312393 >> 172.21.15.195:0/3063312393 conn(0x7f39cc0f0b70 msgr2=0x7f39cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 -- 172.21.15.195:0/3063312393 shutdown_connections 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.855+0000 7f39d1f8f700 1 -- 172.21.15.195:0/3063312393 wait complete. 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39d1f8f700 1 Processor -- start 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39d1f8f700 1 -- start start 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39d1f8f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39d1f8f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc105bd0 con 0x7f39cc0f5410 2024-02-20T00:28:13.874 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39cb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39cb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36556/0 (socket says 172.21.15.195:36556) 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.856+0000 7f39cb7fe700 1 -- 172.21.15.195:0/1193709655 learned_addr learned my addr 172.21.15.195:0/1193709655 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39cb7fe700 1 -- 172.21.15.195:0/1193709655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc005040 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39cb7fe700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f39bc005010 tx=0x7f39bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39bc009320 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc105dd0 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39bc013380 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39bc014e50 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.857+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc106260 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.858+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39bc014420 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.858+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc045ad0 con 0x7f39cc0f5410 2024-02-20T00:28:13.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.858+0000 7f39c3fff700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 0x7f39b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:13.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.858+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39bc040e60 con 0x7f39cc0f5410 2024-02-20T00:28:13.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.858+0000 7f39caffd700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 0x7f39b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:13.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.859+0000 7f39caffd700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 0x7f39b80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f39b40096a0 tx=0x7f39b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:13.876 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:13.863+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39bc0095f0 con 0x7f39cc0f5410 2024-02-20T00:28:14.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.067+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39cc107110 con 0x7f39cc0f5410 2024-02-20T00:28:14.069 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.067+0000 7f39c3fff700 1 -- 172.21.15.195:0/1193709655 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39bc01db70 con 0x7f39cc0f5410 2024-02-20T00:28:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:14.069 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:14.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 msgr2=0x7f39b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:14.070 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 0x7f39b80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f39b40096a0 tx=0x7f39b4006b40).stop 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 msgr2=0x7f39cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f39bc005010 tx=0x7f39bc00b040).stop 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 shutdown_connections 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f39b802ff10 0x7f39b80323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 --2- 172.21.15.195:0/1193709655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39cc0f5410 0x7f39cc1056e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 >> 172.21.15.195:0/1193709655 conn(0x7f39cc0f0b70 msgr2=0x7f39cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 shutdown_connections 2024-02-20T00:28:14.071 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:14.069+0000 7f39d1f8f700 1 -- 172.21.15.195:0/1193709655 wait complete. 2024-02-20T00:28:14.072 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:12.545606+0000 mgr.smithi087.iydsqo (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:14.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:14.068615+0000 mon.smithi087 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.195:0/1193709655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:15.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:15.726 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:14.545963+0000 mgr.smithi087.iydsqo (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/3568493318 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 msgr2=0x7ff2e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/3568493318 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e80f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff2d8009fb0 tx=0x7ff2d8009b30).stop 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/3568493318 shutdown_connections 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/3568493318 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e80f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/3568493318 >> 172.21.15.195:0/3568493318 conn(0x7ff2e80f1c70 msgr2=0x7ff2e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/3568493318 shutdown_connections 2024-02-20T00:28:17.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/3568493318 wait complete. 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 Processor -- start 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.637+0000 7ff2ed0fc700 1 -- start start 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2ed0fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2ed0fc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8069e20 con 0x7ff2e80f6510 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2e6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2e6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49770/0 (socket says 172.21.15.195:49770) 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2e6d9d700 1 -- 172.21.15.195:0/2691592845 learned_addr learned my addr 172.21.15.195:0/2691592845 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2e6d9d700 1 -- 172.21.15.195:0/2691592845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8005040 con 0x7ff2e80f6510 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.638+0000 7ff2e6d9d700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff2d8005010 tx=0x7ff2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8009320 con 0x7ff2e80f6510 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e8069fc0 con 0x7ff2e80f6510 2024-02-20T00:28:17.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2d8013380 con 0x7ff2e80f6510 2024-02-20T00:28:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8014e50 con 0x7ff2e80f6510 2024-02-20T00:28:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e806a450 con 0x7ff2e80f6510 2024-02-20T00:28:17.642 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.639+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2d80263a0 con 0x7ff2e80f6510 2024-02-20T00:28:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.640+0000 7ff2df7fe700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 0x7ff2d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.640+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2d8040ec0 con 0x7ff2e80f6510 2024-02-20T00:28:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.640+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e8059100 con 0x7ff2e80f6510 2024-02-20T00:28:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.640+0000 7ff2e659c700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 0x7ff2d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:17.643 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.640+0000 7ff2e659c700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 0x7ff2d40323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff2d00096a0 tx=0x7ff2d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:17.646 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.644+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2d8012920 con 0x7ff2e80f6510 2024-02-20T00:28:17.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.848+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2e806ab50 con 0x7ff2e80f6510 2024-02-20T00:28:17.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.849+0000 7ff2df7fe700 1 -- 172.21.15.195:0/2691592845 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2d8012330 con 0x7ff2e80f6510 2024-02-20T00:28:17.851 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:17.851 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 msgr2=0x7ff2d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 0x7ff2d40323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff2d00096a0 tx=0x7ff2d0006b40).stop 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 msgr2=0x7ff2e806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff2d8005010 tx=0x7ff2d800b040).stop 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 shutdown_connections 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2d402ff10 0x7ff2d40323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 --2- 172.21.15.195:0/2691592845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2e80f6510 0x7ff2e806b9a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.850+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 >> 172.21.15.195:0/2691592845 conn(0x7ff2e80f1c70 msgr2=0x7ff2e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.851+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 shutdown_connections 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:17.851+0000 7ff2ed0fc700 1 -- 172.21.15.195:0/2691592845 wait complete. 2024-02-20T00:28:17.853 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:16.546327+0000 mgr.smithi087.iydsqo (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:18.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:17.850074+0000 mon.smithi087 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.195:0/2691592845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:19.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:19.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:20.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:18.546621+0000 mgr.smithi087.iydsqo (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 -- 172.21.15.195:0/460292548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 msgr2=0x7f3c2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 --2- 172.21.15.195:0/460292548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3c20009fb0 tx=0x7f3c20009b30).stop 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 -- 172.21.15.195:0/460292548 shutdown_connections 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 --2- 172.21.15.195:0/460292548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 -- 172.21.15.195:0/460292548 >> 172.21.15.195:0/460292548 conn(0x7f3c2c0f0b70 msgr2=0x7f3c2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 -- 172.21.15.195:0/460292548 shutdown_connections 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 -- 172.21.15.195:0/460292548 wait complete. 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.258+0000 7f3c32b81700 1 Processor -- start 2024-02-20T00:28:21.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c32b81700 1 -- start start 2024-02-20T00:28:21.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c32b81700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:21.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c32b81700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c060220 con 0x7f3c2c0f5410 2024-02-20T00:28:21.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c3091d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:21.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c3091d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49782/0 (socket says 172.21.15.195:49782) 2024-02-20T00:28:21.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c3091d700 1 -- 172.21.15.195:0/4019436392 learned_addr learned my addr 172.21.15.195:0/4019436392 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:21.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c3091d700 1 -- 172.21.15.195:0/4019436392 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20005040 con 0x7f3c2c0f5410 2024-02-20T00:28:21.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.259+0000 7f3c3091d700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3c20005010 tx=0x7f3c2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.260+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c20009320 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.260+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c060420 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.260+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c20013380 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.260+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c20014e50 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.260+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c109bb0 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c200095f0 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c059100 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c297fa700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 0x7f3c1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c20040e90 con 0x7f3c2c0f5410 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c2bfff700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 0x7f3c1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:21.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.261+0000 7f3c2bfff700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 0x7f3c1c0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f3c180096a0 tx=0x7f3c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:21.267 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.265+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c20014420 con 0x7f3c2c0f5410 2024-02-20T00:28:21.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.470+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c2c0fafa0 con 0x7f3c2c0f5410 2024-02-20T00:28:21.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.470+0000 7f3c297fa700 1 -- 172.21.15.195:0/4019436392 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c20010230 con 0x7f3c2c0f5410 2024-02-20T00:28:21.472 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:21.472 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 msgr2=0x7f3c1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 0x7f3c1c0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f3c180096a0 tx=0x7f3c18006b40).stop 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 msgr2=0x7f3c2c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3c20005010 tx=0x7f3c2000b040).stop 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 shutdown_connections 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3c1c02ff10 0x7f3c1c0323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 --2- 172.21.15.195:0/4019436392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3c2c0f5410 0x7f3c2c109500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 >> 172.21.15.195:0/4019436392 conn(0x7f3c2c0f0b70 msgr2=0x7f3c2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 shutdown_connections 2024-02-20T00:28:21.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:21.472+0000 7f3c32b81700 1 -- 172.21.15.195:0/4019436392 wait complete. 2024-02-20T00:28:21.475 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:22.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:20.546926+0000 mgr.smithi087.iydsqo (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:21.471582+0000 mon.smithi087 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.195:0/4019436392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:23.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:23.095 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:24.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:22.547256+0000 mgr.smithi087.iydsqo (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 -- 172.21.15.195:0/916790592 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 msgr2=0x7f34440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 --2- 172.21.15.195:0/916790592 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34440f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f3434009fb0 tx=0x7f3434009b30).stop 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 -- 172.21.15.195:0/916790592 shutdown_connections 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 --2- 172.21.15.195:0/916790592 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34440f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 -- 172.21.15.195:0/916790592 >> 172.21.15.195:0/916790592 conn(0x7f34440f0b70 msgr2=0x7f34440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 -- 172.21.15.195:0/916790592 shutdown_connections 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.937+0000 7f344a7c7700 1 -- 172.21.15.195:0/916790592 wait complete. 2024-02-20T00:28:24.941 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f344a7c7700 1 Processor -- start 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f344a7c7700 1 -- start start 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f344a7c7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f344a7c7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3444105bd0 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f3443fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f3443fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49802/0 (socket says 172.21.15.195:49802) 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.938+0000 7f3443fff700 1 -- 172.21.15.195:0/3931325436 learned_addr learned my addr 172.21.15.195:0/3931325436 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f3443fff700 1 -- 172.21.15.195:0/3931325436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434005040 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f3443fff700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3434005010 tx=0x7f343400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3434009320 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3444105dd0 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3434013380 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3434014e50 con 0x7f34440f5410 2024-02-20T00:28:24.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.939+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3444106260 con 0x7f34440f5410 2024-02-20T00:28:24.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.940+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3434014420 con 0x7f34440f5410 2024-02-20T00:28:24.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.940+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3444045ad0 con 0x7f34440f5410 2024-02-20T00:28:24.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.940+0000 7f3440ff9700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 0x7f342c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:24.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.940+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3434040f40 con 0x7f34440f5410 2024-02-20T00:28:24.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.940+0000 7f34437fe700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 0x7f342c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:24.944 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.941+0000 7f34437fe700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 0x7f342c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:24.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:24.945+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34340095f0 con 0x7f34440f5410 2024-02-20T00:28:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.151+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3444107110 con 0x7f34440f5410 2024-02-20T00:28:25.154 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.152+0000 7f3440ff9700 1 -- 172.21.15.195:0/3931325436 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f343401db70 con 0x7f34440f5410 2024-02-20T00:28:25.154 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:25.154 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 msgr2=0x7f342c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 0x7f342c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).stop 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 msgr2=0x7f34441056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3434005010 tx=0x7f343400b040).stop 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 shutdown_connections 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f342c02ff10 0x7f342c0323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 --2- 172.21.15.195:0/3931325436 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34440f5410 0x7f34441056e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 >> 172.21.15.195:0/3931325436 conn(0x7f34440f0b70 msgr2=0x7f34440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 shutdown_connections 2024-02-20T00:28:25.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:25.154+0000 7f344a7c7700 1 -- 172.21.15.195:0/3931325436 wait complete. 2024-02-20T00:28:25.157 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:25.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:25 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:25.153265+0000 mon.smithi087 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.195:0/3931325436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:24.547637+0000 mgr.smithi087.iydsqo (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:26.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:26.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:28.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:26.547962+0000 mgr.smithi087.iydsqo (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:28.671 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.625+0000 7f9e513fb700 1 -- 172.21.15.195:0/2316309155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 msgr2=0x7f9e4c0f6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 --2- 172.21.15.195:0/2316309155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c0f6700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9e40009fb0 tx=0x7f9e40009b30).stop 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- 172.21.15.195:0/2316309155 shutdown_connections 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 --2- 172.21.15.195:0/2316309155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c0f6700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- 172.21.15.195:0/2316309155 >> 172.21.15.195:0/2316309155 conn(0x7f9e4c0f1a60 msgr2=0x7f9e4c0f3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- 172.21.15.195:0/2316309155 shutdown_connections 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- 172.21.15.195:0/2316309155 wait complete. 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 Processor -- start 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- start start 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.626+0000 7f9e513fb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c106b50 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e4affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:28.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e4affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60198/0 (socket says 172.21.15.195:60198) 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e4affd700 1 -- 172.21.15.195:0/3889150740 learned_addr learned my addr 172.21.15.195:0/3889150740 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e4affd700 1 -- 172.21.15.195:0/3889150740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e40005040 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e4affd700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9e40005010 tx=0x7f9e4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e40009320 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c106d50 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e40013380 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e40014e50 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.627+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c1071e0 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.628+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e400095f0 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.628+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c045ad0 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.628+0000 7f9e337fe700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 0x7f9e340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.628+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e40040e90 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.629+0000 7f9e4a7fc700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 0x7f9e340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.629+0000 7f9e4a7fc700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 0x7f9e340323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9e3c0096a0 tx=0x7f9e3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:28.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.633+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e40014420 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.840 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.838+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e4c107fe0 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.840 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e337fe700 1 -- 172.21.15.195:0/3889150740 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e40010230 con 0x7f9e4c0f62e0 2024-02-20T00:28:28.841 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:28.841 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:28.842 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 msgr2=0x7f9e340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:28.842 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 0x7f9e340323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9e3c0096a0 tx=0x7f9e3c006b40).stop 2024-02-20T00:28:28.842 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 msgr2=0x7f9e4c106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9e40005010 tx=0x7f9e4000b040).stop 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 shutdown_connections 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9e3402ff10 0x7f9e340323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 --2- 172.21.15.195:0/3889150740 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e4c0f62e0 0x7f9e4c106660 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 >> 172.21.15.195:0/3889150740 conn(0x7f9e4c0f1a60 msgr2=0x7f9e4c0f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 shutdown_connections 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:28.840+0000 7f9e513fb700 1 -- 172.21.15.195:0/3889150740 wait complete. 2024-02-20T00:28:28.843 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:29.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:28.839979+0000 mon.smithi087 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.195:0/3889150740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:30.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:30.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:30.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:28.548278+0000 mgr.smithi087.iydsqo (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 -- 172.21.15.195:0/72056386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 msgr2=0x7ff2fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 --2- 172.21.15.195:0/72056386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc0f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff2f0009fb0 tx=0x7ff2f0009b30).stop 2024-02-20T00:28:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 -- 172.21.15.195:0/72056386 shutdown_connections 2024-02-20T00:28:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 --2- 172.21.15.195:0/72056386 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc0f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:32.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 -- 172.21.15.195:0/72056386 >> 172.21.15.195:0/72056386 conn(0x7ff2fc0f0b70 msgr2=0x7ff2fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 -- 172.21.15.195:0/72056386 shutdown_connections 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.308+0000 7ff3040a0700 1 -- 172.21.15.195:0/72056386 wait complete. 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff3040a0700 1 Processor -- start 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff3040a0700 1 -- start start 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff3040a0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff3040a0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc108ea0 con 0x7ff2fc0f5410 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff301e3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:32.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff301e3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60218/0 (socket says 172.21.15.195:60218) 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff301e3c700 1 -- 172.21.15.195:0/2584896996 learned_addr learned my addr 172.21.15.195:0/2584896996 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff301e3c700 1 -- 172.21.15.195:0/2584896996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0005040 con 0x7ff2fc0f5410 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff301e3c700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff2f0005010 tx=0x7ff2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.309+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f0009320 con 0x7ff2fc0f5410 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc1090a0 con 0x7ff2fc0f5410 2024-02-20T00:28:32.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f0013380 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f0014e50 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc109530 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2f00263a0 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc045ad0 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.310+0000 7ff2ee7fc700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 0x7ff2e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.311+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2f0041250 con 0x7ff2fc0f5410 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.311+0000 7ff30163b700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 0x7ff2e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:32.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.311+0000 7ff30163b700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 0x7ff2e80323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff2f80096a0 tx=0x7ff2f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:32.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.316+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2f001cd00 con 0x7ff2fc0f5410 2024-02-20T00:28:32.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.519+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2fc0fafa0 con 0x7ff2fc0f5410 2024-02-20T00:28:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.519+0000 7ff2ee7fc700 1 -- 172.21.15.195:0/2584896996 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2f001cb20 con 0x7ff2fc0f5410 2024-02-20T00:28:32.521 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:32.521 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 msgr2=0x7ff2e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 0x7ff2e80323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff2f80096a0 tx=0x7ff2f8006b40).stop 2024-02-20T00:28:32.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 msgr2=0x7ff2fc1089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff2f0005010 tx=0x7ff2f000b040).stop 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 shutdown_connections 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff2e802ff10 0x7ff2e80323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 --2- 172.21.15.195:0/2584896996 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2fc0f5410 0x7ff2fc1089b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 >> 172.21.15.195:0/2584896996 conn(0x7ff2fc0f0b70 msgr2=0x7ff2fc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 shutdown_connections 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:32.522+0000 7ff3040a0700 1 -- 172.21.15.195:0/2584896996 wait complete. 2024-02-20T00:28:32.524 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:32.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:30.548572+0000 mgr.smithi087.iydsqo (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:33.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:32.520557+0000 mon.smithi087 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.195:0/2584896996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:34.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:34.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:32.548869+0000 mgr.smithi087.iydsqo (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- 172.21.15.195:0/767797070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 msgr2=0x7f19d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 --2- 172.21.15.195:0/767797070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d80f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f19cc009fb0 tx=0x7f19cc009b30).stop 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- 172.21.15.195:0/767797070 shutdown_connections 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 --2- 172.21.15.195:0/767797070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d80f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- 172.21.15.195:0/767797070 >> 172.21.15.195:0/767797070 conn(0x7f19d80f09a0 msgr2=0x7f19d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:36.020 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- 172.21.15.195:0/767797070 shutdown_connections 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- 172.21.15.195:0/767797070 wait complete. 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 Processor -- start 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- start start 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:35.999+0000 7f19d759e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8060280 con 0x7f19d80f51a0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60242/0 (socket says 172.21.15.195:60242) 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d659c700 1 -- 172.21.15.195:0/1888235063 learned_addr learned my addr 172.21.15.195:0/1888235063 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d659c700 1 -- 172.21.15.195:0/1888235063 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc005040 con 0x7f19d80f51a0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d659c700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f19cc0089f0 tx=0x7f19cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19cc009320 con 0x7f19d80f51a0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d8060480 con 0x7f19d80f51a0 2024-02-20T00:28:36.021 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19cc013380 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19cc014e20 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.000+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d81099c0 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.001+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19cc0095f0 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.001+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8045ad0 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.001+0000 7f19c6ffd700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 0x7f19c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.001+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19cc040e90 con 0x7f19d80f51a0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.001+0000 7f19d5d9b700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 0x7f19c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.002+0000 7f19d5d9b700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 0x7f19c00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f19c80096a0 tx=0x7f19c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:36.022 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.006+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19cc014420 con 0x7f19d80f51a0 2024-02-20T00:28:36.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.216+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19d80fad30 con 0x7f19d80f51a0 2024-02-20T00:28:36.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.218+0000 7f19c6ffd700 1 -- 172.21.15.195:0/1888235063 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19cc01d980 con 0x7f19d80f51a0 2024-02-20T00:28:36.218 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:36.219 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:36.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 msgr2=0x7f19c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 0x7f19c00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f19c80096a0 tx=0x7f19c8006b40).stop 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 msgr2=0x7f19d8109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f19cc0089f0 tx=0x7f19cc00b040).stop 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 shutdown_connections 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f19c002ff10 0x7f19c00323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 --2- 172.21.15.195:0/1888235063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19d80f51a0 0x7f19d8109290 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 >> 172.21.15.195:0/1888235063 conn(0x7f19d80f09a0 msgr2=0x7f19d80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 shutdown_connections 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:36.219+0000 7f19d759e700 1 -- 172.21.15.195:0/1888235063 wait complete. 2024-02-20T00:28:36.221 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:36.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:34.549177+0000 mgr.smithi087.iydsqo (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:37.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:36.217745+0000 mon.smithi087 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.195:0/1888235063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:37.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:37.872 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:38.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:36.549539+0000 mgr.smithi087.iydsqo (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 -- 172.21.15.195:0/3354422429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 msgr2=0x7f36d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 --2- 172.21.15.195:0/3354422429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d80f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f36d4009fb0 tx=0x7f36d4009b30).stop 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 -- 172.21.15.195:0/3354422429 shutdown_connections 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 --2- 172.21.15.195:0/3354422429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d80f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 -- 172.21.15.195:0/3354422429 >> 172.21.15.195:0/3354422429 conn(0x7f36d80f1c70 msgr2=0x7f36d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 -- 172.21.15.195:0/3354422429 shutdown_connections 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.671+0000 7f36e0279700 1 -- 172.21.15.195:0/3354422429 wait complete. 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36e0279700 1 Processor -- start 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36e0279700 1 -- start start 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36e0279700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:39.675 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36e0279700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8069e20 con 0x7f36d80f6510 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36de015700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36de015700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38930/0 (socket says 172.21.15.195:38930) 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36de015700 1 -- 172.21.15.195:0/1999591664 learned_addr learned my addr 172.21.15.195:0/1999591664 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.672+0000 7f36de015700 1 -- 172.21.15.195:0/1999591664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4005040 con 0x7f36d80f6510 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36de015700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f36d4005010 tx=0x7f36d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:39.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4009320 con 0x7f36d80f6510 2024-02-20T00:28:39.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d8069fc0 con 0x7f36d80f6510 2024-02-20T00:28:39.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36d4013380 con 0x7f36d80f6510 2024-02-20T00:28:39.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4014e50 con 0x7f36d80f6510 2024-02-20T00:28:39.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.673+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d806a450 con 0x7f36d80f6510 2024-02-20T00:28:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d40095f0 con 0x7f36d80f6510 2024-02-20T00:28:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d8059100 con 0x7f36d80f6510 2024-02-20T00:28:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36ceffd700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 0x7f36c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d4040e90 con 0x7f36d80f6510 2024-02-20T00:28:39.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36dd814700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 0x7f36c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:39.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.674+0000 7f36dd814700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 0x7f36c40323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f36c80096a0 tx=0x7f36c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:39.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.678+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d4014420 con 0x7f36d80f6510 2024-02-20T00:28:39.888 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.883+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36d80fc0a0 con 0x7f36d80f6510 2024-02-20T00:28:39.888 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.884+0000 7f36ceffd700 1 -- 172.21.15.195:0/1999591664 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36d4010230 con 0x7f36d80f6510 2024-02-20T00:28:39.888 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:39.888 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 msgr2=0x7f36c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 0x7f36c40323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f36c80096a0 tx=0x7f36c8006b40).stop 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 msgr2=0x7f36d806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f36d4005010 tx=0x7f36d400b040).stop 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 shutdown_connections 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36c402ff10 0x7f36c40323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:39.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 --2- 172.21.15.195:0/1999591664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36d80f6510 0x7f36d806b9a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:39.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 >> 172.21.15.195:0/1999591664 conn(0x7f36d80f1c70 msgr2=0x7f36d80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:39.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 shutdown_connections 2024-02-20T00:28:39.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:39.885+0000 7f36e0279700 1 -- 172.21.15.195:0/1999591664 wait complete. 2024-02-20T00:28:39.890 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:40.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:38.549903+0000 mgr.smithi087.iydsqo (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:40.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:39.884751+0000 mon.smithi087 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.195:0/1999591664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:41.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:41.473 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:42.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:40.550272+0000 mgr.smithi087.iydsqo (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:43.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3314593915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 msgr2=0x7f40a80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:43.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3314593915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a80f9150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f4098009fb0 tx=0x7f4098009b30).stop 2024-02-20T00:28:43.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3314593915 shutdown_connections 2024-02-20T00:28:43.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3314593915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a80f9150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3314593915 >> 172.21.15.195:0/3314593915 conn(0x7f40a80f0b70 msgr2=0x7f40a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3314593915 shutdown_connections 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3314593915 wait complete. 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 Processor -- start 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- start start 2024-02-20T00:28:43.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.279+0000 7f40ad6d7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a8106cb0 con 0x7f40a8060220 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40a6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40a6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38948/0 (socket says 172.21.15.195:38948) 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40a6ffd700 1 -- 172.21.15.195:0/3913693635 learned_addr learned my addr 172.21.15.195:0/3913693635 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40a6ffd700 1 -- 172.21.15.195:0/3913693635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098005040 con 0x7f40a8060220 2024-02-20T00:28:43.287 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40a6ffd700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f40980089f0 tx=0x7f409800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4098009320 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a8106eb0 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4098013380 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4098014e20 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.280+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a8107340 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.281+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4098014420 con 0x7f40a8060220 2024-02-20T00:28:43.288 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.281+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a8045ad0 con 0x7f40a8060220 2024-02-20T00:28:43.289 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.281+0000 7f408f7fe700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 0x7f40900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:43.289 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.281+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4098040f50 con 0x7f40a8060220 2024-02-20T00:28:43.289 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.282+0000 7f40a67fc700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 0x7f40900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:43.289 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.282+0000 7f40a67fc700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 0x7f40900323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:43.290 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.286+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f409806d2b0 con 0x7f40a8060220 2024-02-20T00:28:43.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.490+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40a81081f0 con 0x7f40a8060220 2024-02-20T00:28:43.493 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.491+0000 7f408f7fe700 1 -- 172.21.15.195:0/3913693635 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40980095f0 con 0x7f40a8060220 2024-02-20T00:28:43.494 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:43.494 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:43.494 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 msgr2=0x7f40900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 0x7f40900323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).stop 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 msgr2=0x7f40a81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f40980089f0 tx=0x7f409800b040).stop 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 shutdown_connections 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f409002ff10 0x7f40900323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 --2- 172.21.15.195:0/3913693635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40a8060220 0x7f40a81067c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 >> 172.21.15.195:0/3913693635 conn(0x7f40a80f0b70 msgr2=0x7f40a80f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 shutdown_connections 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:43.493+0000 7f40ad6d7700 1 -- 172.21.15.195:0/3913693635 wait complete. 2024-02-20T00:28:43.495 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:42.550557+0000 mgr.smithi087.iydsqo (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:44.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:43.492470+0000 mon.smithi087 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.195:0/3913693635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:45.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:45.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:44.550895+0000 mgr.smithi087.iydsqo (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 -- 172.21.15.195:0/4169373323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 msgr2=0x7f06600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 --2- 172.21.15.195:0/4169373323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06600f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0654009fb0 tx=0x7f0654009b30).stop 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 -- 172.21.15.195:0/4169373323 shutdown_connections 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 --2- 172.21.15.195:0/4169373323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06600f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 -- 172.21.15.195:0/4169373323 >> 172.21.15.195:0/4169373323 conn(0x7f06600f1c70 msgr2=0x7f06600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:47.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 -- 172.21.15.195:0/4169373323 shutdown_connections 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.215+0000 7f0664efa700 1 -- 172.21.15.195:0/4169373323 wait complete. 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f0664efa700 1 Processor -- start 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f0664efa700 1 -- start start 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f0664efa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f0664efa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660106cd0 con 0x7f06600f6510 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f065e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f065e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36612/0 (socket says 172.21.15.195:36612) 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f065e59c700 1 -- 172.21.15.195:0/2119794397 learned_addr learned my addr 172.21.15.195:0/2119794397 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.216+0000 7f065e59c700 1 -- 172.21.15.195:0/2119794397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654005040 con 0x7f06600f6510 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f065e59c700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0654005010 tx=0x7f065400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:47.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0654009320 con 0x7f06600f6510 2024-02-20T00:28:47.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0660106ed0 con 0x7f06600f6510 2024-02-20T00:28:47.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0654013380 con 0x7f06600f6510 2024-02-20T00:28:47.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0654014e50 con 0x7f06600f6510 2024-02-20T00:28:47.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.217+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0660107360 con 0x7f06600f6510 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.218+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06540095f0 con 0x7f06600f6510 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.218+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660045ad0 con 0x7f06600f6510 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.218+0000 7f064effd700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.218+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0654040e90 con 0x7f06600f6510 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.218+0000 7f065dd9b700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:47.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.219+0000 7f065dd9b700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 0x7f06480323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f06500096a0 tx=0x7f0650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:47.224 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.223+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0654014420 con 0x7f06600f6510 2024-02-20T00:28:47.430 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.428+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0660108210 con 0x7f06600f6510 2024-02-20T00:28:47.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.429+0000 7f064effd700 1 -- 172.21.15.195:0/2119794397 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0654010230 con 0x7f06600f6510 2024-02-20T00:28:47.431 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:47.431 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:47.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 msgr2=0x7f06480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:47.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 0x7f06480323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f06500096a0 tx=0x7f0650006b40).stop 2024-02-20T00:28:47.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 msgr2=0x7f06601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:47.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0654005010 tx=0x7f065400b040).stop 2024-02-20T00:28:47.434 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 shutdown_connections 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 --2- 172.21.15.195:0/2119794397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06600f6510 0x7f06601067e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 >> 172.21.15.195:0/2119794397 conn(0x7f06600f1c70 msgr2=0x7f06600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 shutdown_connections 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:47.431+0000 7f0664efa700 1 -- 172.21.15.195:0/2119794397 wait complete. 2024-02-20T00:28:47.435 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:48.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:46.551131+0000 mgr.smithi087.iydsqo (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:48.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:47.430363+0000 mon.smithi087 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.195:0/2119794397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:49.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:49.069 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:50.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:48.551446+0000 mgr.smithi087.iydsqo (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:50.866 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- 172.21.15.195:0/101753493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 msgr2=0x7f44c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 --2- 172.21.15.195:0/101753493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c00f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f44b0009fb0 tx=0x7f44b0009b30).stop 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- 172.21.15.195:0/101753493 shutdown_connections 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 --2- 172.21.15.195:0/101753493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c00f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- 172.21.15.195:0/101753493 >> 172.21.15.195:0/101753493 conn(0x7f44c00f0b70 msgr2=0x7f44c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- 172.21.15.195:0/101753493 shutdown_connections 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- 172.21.15.195:0/101753493 wait complete. 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 Processor -- start 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- start start 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.864+0000 7f44c5db6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0105c10 con 0x7f44c00f5410 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36628/0 (socket says 172.21.15.195:36628) 2024-02-20T00:28:50.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44bf7fe700 1 -- 172.21.15.195:0/85611313 learned_addr learned my addr 172.21.15.195:0/85611313 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44bf7fe700 1 -- 172.21.15.195:0/85611313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0005040 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44bf7fe700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f44b0005010 tx=0x7f44b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44b0009320 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c0105e10 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44b0013380 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.865+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44b0014e50 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.866+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c01062a0 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.866+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44b00263a0 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.866+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0045ad0 con 0x7f44c00f5410 2024-02-20T00:28:50.869 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.866+0000 7f44a7fff700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 0x7f44a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.866+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44b0040f60 con 0x7f44c00f5410 2024-02-20T00:28:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.867+0000 7f44beffd700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 0x7f44a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:50.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.867+0000 7f44beffd700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 0x7f44a80323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f44b40096a0 tx=0x7f44b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:50.872 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:50.871+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44b00126f0 con 0x7f44c00f5410 2024-02-20T00:28:51.076 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.074+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44c0107150 con 0x7f44c00f5410 2024-02-20T00:28:51.077 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.075+0000 7f44a7fff700 1 -- 172.21.15.195:0/85611313 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44b0012330 con 0x7f44c00f5410 2024-02-20T00:28:51.077 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:51.077 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:51.078 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.076+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 msgr2=0x7f44a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.076+0000 7f44c5db6700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 0x7f44a80323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f44b40096a0 tx=0x7f44b4006b40).stop 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.076+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 msgr2=0x7f44c0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.076+0000 7f44c5db6700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f44b0005010 tx=0x7f44b000b040).stop 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 shutdown_connections 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f44a802ff10 0x7f44a80323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 --2- 172.21.15.195:0/85611313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44c00f5410 0x7f44c0105720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 >> 172.21.15.195:0/85611313 conn(0x7f44c00f0b70 msgr2=0x7f44c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 shutdown_connections 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:51.077+0000 7f44c5db6700 1 -- 172.21.15.195:0/85611313 wait complete. 2024-02-20T00:28:51.079 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:51.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:51 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:51.076199+0000 mon.smithi087 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.195:0/85611313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:52.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:50.551735+0000 mgr.smithi087.iydsqo (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:52.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:52.749 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:54.482 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 -- 172.21.15.195:0/3459384331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 msgr2=0x7efff40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:54.482 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 --2- 172.21.15.195:0/3459384331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff40f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7effe4009fb0 tx=0x7effe4009b30).stop 2024-02-20T00:28:54.482 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 -- 172.21.15.195:0/3459384331 shutdown_connections 2024-02-20T00:28:54.482 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 --2- 172.21.15.195:0/3459384331 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff40f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 -- 172.21.15.195:0/3459384331 >> 172.21.15.195:0/3459384331 conn(0x7efff40f1c70 msgr2=0x7efff40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 -- 172.21.15.195:0/3459384331 shutdown_connections 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.479+0000 7efffb950700 1 -- 172.21.15.195:0/3459384331 wait complete. 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efffb950700 1 Processor -- start 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efffb950700 1 -- start start 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efffb950700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efffb950700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff4106cd0 con 0x7efff40f6510 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efff96ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efff96ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36640/0 (socket says 172.21.15.195:36640) 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.480+0000 7efff96ec700 1 -- 172.21.15.195:0/4148979904 learned_addr learned my addr 172.21.15.195:0/4148979904 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7efff96ec700 1 -- 172.21.15.195:0/4148979904 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4005040 con 0x7efff40f6510 2024-02-20T00:28:54.483 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7efff96ec700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7effe4005010 tx=0x7effe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe4009320 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff4106ed0 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effe4013380 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe4014e50 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.481+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff4107360 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.482+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effe40095f0 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.482+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff4045ad0 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.482+0000 7effe9ffb700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 0x7effe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.482+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effe4040e60 con 0x7efff40f6510 2024-02-20T00:28:54.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.482+0000 7efff8eeb700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 0x7effe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:54.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.483+0000 7efff8eeb700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 0x7effe00323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:54.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.487+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effe4014420 con 0x7efff40f6510 2024-02-20T00:28:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:52.552030+0000 mgr.smithi087.iydsqo (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:54.694 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.692+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efff4108210 con 0x7efff40f6510 2024-02-20T00:28:54.695 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.693+0000 7effe9ffb700 1 -- 172.21.15.195:0/4148979904 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effe401d980 con 0x7efff40f6510 2024-02-20T00:28:54.695 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:54.695 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:54.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 msgr2=0x7effe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:54.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 0x7effe00323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).stop 2024-02-20T00:28:54.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 msgr2=0x7efff41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:54.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7effe4005010 tx=0x7effe400b040).stop 2024-02-20T00:28:54.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 shutdown_connections 2024-02-20T00:28:54.698 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effe002ff10 0x7effe00323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:54.698 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 --2- 172.21.15.195:0/4148979904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efff40f6510 0x7efff41067e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:54.698 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 >> 172.21.15.195:0/4148979904 conn(0x7efff40f1c70 msgr2=0x7efff40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:54.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 shutdown_connections 2024-02-20T00:28:54.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:54.694+0000 7efffb950700 1 -- 172.21.15.195:0/4148979904 wait complete. 2024-02-20T00:28:54.699 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:55.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:55 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:54.694103+0000 mon.smithi087 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.195:0/4148979904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:56.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:56.287 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:28:56.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:54.552350+0000 mgr.smithi087.iydsqo (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:56.666381+0000 mon.smithi087 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:28:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:56.667977+0000 mon.smithi087 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 -- 172.21.15.195:0/1264640615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 msgr2=0x7f8e500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1264640615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e500f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8e38009fb0 tx=0x7f8e38009b30).stop 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 -- 172.21.15.195:0/1264640615 shutdown_connections 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1264640615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e500f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 -- 172.21.15.195:0/1264640615 >> 172.21.15.195:0/1264640615 conn(0x7f8e500f0b70 msgr2=0x7f8e500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 -- 172.21.15.195:0/1264640615 shutdown_connections 2024-02-20T00:28:58.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.036+0000 7f8e5543a700 1 -- 172.21.15.195:0/1264640615 wait complete. 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e5543a700 1 Processor -- start 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e5543a700 1 -- start start 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e5543a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e5543a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50106d00 con 0x7f8e500f5410 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e4effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e4effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38610/0 (socket says 172.21.15.195:38610) 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.037+0000 7f8e4effd700 1 -- 172.21.15.195:0/1149184609 learned_addr learned my addr 172.21.15.195:0/1149184609 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e4effd700 1 -- 172.21.15.195:0/1149184609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38005040 con 0x7f8e500f5410 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e4effd700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8e38005010 tx=0x7f8e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e38009320 con 0x7f8e500f5410 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e38013380 con 0x7f8e500f5410 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e50106f00 con 0x7f8e500f5410 2024-02-20T00:28:58.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e38014e50 con 0x7f8e500f5410 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.038+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e50107390 con 0x7f8e500f5410 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.039+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e380263a0 con 0x7f8e500f5410 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.039+0000 7f8e477fe700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.039+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e38041f10 con 0x7f8e500f5410 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.039+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50059100 con 0x7f8e500f5410 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.039+0000 7f8e4e7fc700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:28:58.042 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.040+0000 7f8e4e7fc700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8e400096a0 tx=0x7f8e40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:28:58.045 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.044+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e38012920 con 0x7f8e500f5410 2024-02-20T00:28:58.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.246+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e500fafa0 con 0x7f8e500f5410 2024-02-20T00:28:58.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.247+0000 7f8e477fe700 1 -- 172.21.15.195:0/1149184609 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e3802b020 con 0x7f8e500f5410 2024-02-20T00:28:58.249 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:28:58.249 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:28:58.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 msgr2=0x7f8e3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:58.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8e400096a0 tx=0x7f8e40006b40).stop 2024-02-20T00:28:58.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 msgr2=0x7f8e50106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8e38005010 tx=0x7f8e3800b040).stop 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 shutdown_connections 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f8e3c02ff10 0x7f8e3c0323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 --2- 172.21.15.195:0/1149184609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e500f5410 0x7f8e50106810 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 >> 172.21.15.195:0/1149184609 conn(0x7f8e500f0b70 msgr2=0x7f8e5005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 shutdown_connections 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:28:58.249+0000 7f8e5543a700 1 -- 172.21.15.195:0/1149184609 wait complete. 2024-02-20T00:28:58.251 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:28:58.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:56.552706+0000 mgr.smithi087.iydsqo (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:28:59.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:28:59 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:28:58.248320+0000 mon.smithi087 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.195:0/1149184609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:28:59.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:28:59.929 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:00.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:28:58.553006+0000 mgr.smithi087.iydsqo (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- 172.21.15.195:0/3310551418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 msgr2=0x7f25a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 --2- 172.21.15.195:0/3310551418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a80f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2598009fb0 tx=0x7f2598009b30).stop 2024-02-20T00:29:01.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- 172.21.15.195:0/3310551418 shutdown_connections 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 --2- 172.21.15.195:0/3310551418 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a80f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- 172.21.15.195:0/3310551418 >> 172.21.15.195:0/3310551418 conn(0x7f25a80f1c70 msgr2=0x7f25a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- 172.21.15.195:0/3310551418 shutdown_connections 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- 172.21.15.195:0/3310551418 wait complete. 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 Processor -- start 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.857+0000 7f25b0125700 1 -- start start 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25b0125700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25b0125700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8069e20 con 0x7f25a80f6510 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25adec1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25adec1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38616/0 (socket says 172.21.15.195:38616) 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25adec1700 1 -- 172.21.15.195:0/3045904591 learned_addr learned my addr 172.21.15.195:0/3045904591 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25adec1700 1 -- 172.21.15.195:0/3045904591 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598005040 con 0x7f25a80f6510 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25adec1700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f2598005010 tx=0x7f259800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:01.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2598009320 con 0x7f25a80f6510 2024-02-20T00:29:01.862 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a8069fc0 con 0x7f25a80f6510 2024-02-20T00:29:01.862 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2598013380 con 0x7f25a80f6510 2024-02-20T00:29:01.862 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.858+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2598014e50 con 0x7f25a80f6510 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.859+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a806a450 con 0x7f25a80f6510 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.859+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2598014420 con 0x7f25a80f6510 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.859+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a8059100 con 0x7f25a80f6510 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.859+0000 7f259e7fc700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 0x7f25940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.859+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2598040f40 con 0x7f25a80f6510 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.860+0000 7f25ad6c0700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 0x7f25940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:01.863 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.860+0000 7f25ad6c0700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 0x7f25940323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f25a40096a0 tx=0x7f25a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:01.865 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:01.864+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25980095f0 con 0x7f25a80f6510 2024-02-20T00:29:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.070+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25a80fc0a0 con 0x7f25a80f6510 2024-02-20T00:29:02.072 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.071+0000 7f259e7fc700 1 -- 172.21.15.195:0/3045904591 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f259801db70 con 0x7f25a80f6510 2024-02-20T00:29:02.073 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:02.073 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:02.074 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 msgr2=0x7f25940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 0x7f25940323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f25a40096a0 tx=0x7f25a4006b40).stop 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 msgr2=0x7f25a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f2598005010 tx=0x7f259800b040).stop 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 shutdown_connections 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f259402ff10 0x7f25940323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 --2- 172.21.15.195:0/3045904591 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25a80f6510 0x7f25a806b9a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.072+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 >> 172.21.15.195:0/3045904591 conn(0x7f25a80f1c70 msgr2=0x7f25a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.073+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 shutdown_connections 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:02.073+0000 7f25b0125700 1 -- 172.21.15.195:0/3045904591 wait complete. 2024-02-20T00:29:02.075 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:02.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:00.553269+0000 mgr.smithi087.iydsqo (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:02.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:02.071969+0000 mon.smithi087 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.195:0/3045904591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:03.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:03.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:04.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:02.553551+0000 mgr.smithi087.iydsqo (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:05.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 -- 172.21.15.195:0/2915472450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 msgr2=0x7faa5c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 --2- 172.21.15.195:0/2915472450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f90f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7faa44009fb0 tx=0x7faa44009b30).stop 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 -- 172.21.15.195:0/2915472450 shutdown_connections 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 --2- 172.21.15.195:0/2915472450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f90f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 -- 172.21.15.195:0/2915472450 >> 172.21.15.195:0/2915472450 conn(0x7faa5c0f0b70 msgr2=0x7faa5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 -- 172.21.15.195:0/2915472450 shutdown_connections 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.622+0000 7faa61aec700 1 -- 172.21.15.195:0/2915472450 wait complete. 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa61aec700 1 Processor -- start 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa61aec700 1 -- start start 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa61aec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:05.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa61aec700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c0f4ff0 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa5b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa5b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38626/0 (socket says 172.21.15.195:38626) 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa5b7fe700 1 -- 172.21.15.195:0/1334305381 learned_addr learned my addr 172.21.15.195:0/1334305381 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa5b7fe700 1 -- 172.21.15.195:0/1334305381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44005040 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.623+0000 7faa5b7fe700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7faa440089f0 tx=0x7faa4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.624+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa44009320 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.624+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c0f5190 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.624+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa44013380 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.624+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa44014e20 con 0x7faa5c060280 2024-02-20T00:29:05.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.624+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c0f5620 con 0x7faa5c060280 2024-02-20T00:29:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa44014420 con 0x7faa5c060280 2024-02-20T00:29:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c059100 con 0x7faa5c060280 2024-02-20T00:29:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa53fff700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 0x7faa480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa44040f50 con 0x7faa5c060280 2024-02-20T00:29:05.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa5affd700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 0x7faa480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:05.629 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.625+0000 7faa5affd700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 0x7faa480323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7faa4c0096a0 tx=0x7faa4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:05.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.630+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa4406d2b0 con 0x7faa5c060280 2024-02-20T00:29:05.835 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.834+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa5c0fc090 con 0x7faa5c060280 2024-02-20T00:29:05.836 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.834+0000 7faa53fff700 1 -- 172.21.15.195:0/1334305381 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa440095f0 con 0x7faa5c060280 2024-02-20T00:29:05.836 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:05.836 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:05.837 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 msgr2=0x7faa480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 0x7faa480323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7faa4c0096a0 tx=0x7faa4c006b40).stop 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 msgr2=0x7faa5c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7faa440089f0 tx=0x7faa4400b040).stop 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 shutdown_connections 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa4802ff10 0x7faa480323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 --2- 172.21.15.195:0/1334305381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa5c060280 0x7faa5c0f6b70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 >> 172.21.15.195:0/1334305381 conn(0x7faa5c0f0b70 msgr2=0x7faa5c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 shutdown_connections 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:05.836+0000 7faa61aec700 1 -- 172.21.15.195:0/1334305381 wait complete. 2024-02-20T00:29:05.838 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:06.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:04.553906+0000 mgr.smithi087.iydsqo (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:06.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:05.835446+0000 mon.smithi087 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.195:0/1334305381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:07.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:07.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:08.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:06.554247+0000 mgr.smithi087.iydsqo (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 -- 172.21.15.195:0/1915667056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 msgr2=0x7fe0fc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 --2- 172.21.15.195:0/1915667056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f90f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe0f0009fb0 tx=0x7fe0f0009b30).stop 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 -- 172.21.15.195:0/1915667056 shutdown_connections 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 --2- 172.21.15.195:0/1915667056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f90f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 -- 172.21.15.195:0/1915667056 >> 172.21.15.195:0/1915667056 conn(0x7fe0fc0f0b70 msgr2=0x7fe0fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 -- 172.21.15.195:0/1915667056 shutdown_connections 2024-02-20T00:29:09.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 -- 172.21.15.195:0/1915667056 wait complete. 2024-02-20T00:29:09.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.327+0000 7fe10160f700 1 Processor -- start 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe10160f700 1 -- start start 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe10160f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe10160f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0f4ff0 con 0x7fe0fc060280 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe0faffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe0faffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39732/0 (socket says 172.21.15.195:39732) 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe0faffd700 1 -- 172.21.15.195:0/1097874262 learned_addr learned my addr 172.21.15.195:0/1097874262 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.328+0000 7fe0faffd700 1 -- 172.21.15.195:0/1097874262 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0005040 con 0x7fe0fc060280 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe0faffd700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe0f00089f0 tx=0x7fe0f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f0009320 con 0x7fe0fc060280 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc0f5190 con 0x7fe0fc060280 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0f0013380 con 0x7fe0fc060280 2024-02-20T00:29:09.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f0014e20 con 0x7fe0fc060280 2024-02-20T00:29:09.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.329+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc0f5620 con 0x7fe0fc060280 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0f0014420 con 0x7fe0fc060280 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc059100 con 0x7fe0fc060280 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe0e37fe700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 0x7fe0e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0f0040fc0 con 0x7fe0fc060280 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe0fa7fc700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 0x7fe0e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:09.333 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.330+0000 7fe0fa7fc700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 0x7fe0e40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe0ec0096a0 tx=0x7fe0ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:09.336 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.334+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0f006d2b0 con 0x7fe0fc060280 2024-02-20T00:29:09.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.538+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0fc0fc090 con 0x7fe0fc060280 2024-02-20T00:29:09.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.539+0000 7fe0e37fe700 1 -- 172.21.15.195:0/1097874262 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0f00095f0 con 0x7fe0fc060280 2024-02-20T00:29:09.541 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:09.541 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:09.542 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 msgr2=0x7fe0e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 0x7fe0e40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe0ec0096a0 tx=0x7fe0ec006b40).stop 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 msgr2=0x7fe0fc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe0f00089f0 tx=0x7fe0f000b040).stop 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 shutdown_connections 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe0e402ff10 0x7fe0e40323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 --2- 172.21.15.195:0/1097874262 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc060280 0x7fe0fc0f6b70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 >> 172.21.15.195:0/1097874262 conn(0x7fe0fc0f0b70 msgr2=0x7fe0fc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 shutdown_connections 2024-02-20T00:29:09.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:09.541+0000 7fe10160f700 1 -- 172.21.15.195:0/1097874262 wait complete. 2024-02-20T00:29:09.544 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:10.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:08.554505+0000 mgr.smithi087.iydsqo (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:10.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:09.540296+0000 mon.smithi087 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.195:0/1097874262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:11.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:11.179 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:12.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:10.554875+0000 mgr.smithi087.iydsqo (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:13.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 -- 172.21.15.195:0/3714298450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 msgr2=0x7f6ab80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 --2- 172.21.15.195:0/3714298450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab80f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6aac009fb0 tx=0x7f6aac009b30).stop 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 -- 172.21.15.195:0/3714298450 shutdown_connections 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 --2- 172.21.15.195:0/3714298450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab80f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 -- 172.21.15.195:0/3714298450 >> 172.21.15.195:0/3714298450 conn(0x7f6ab80f0b70 msgr2=0x7f6ab80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 -- 172.21.15.195:0/3714298450 shutdown_connections 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.121+0000 7f6abee25700 1 -- 172.21.15.195:0/3714298450 wait complete. 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abee25700 1 Processor -- start 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abee25700 1 -- start start 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abee25700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:13.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abee25700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8060240 con 0x7f6ab80f5410 2024-02-20T00:29:13.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abcbc1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:13.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abcbc1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39742/0 (socket says 172.21.15.195:39742) 2024-02-20T00:29:13.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abcbc1700 1 -- 172.21.15.195:0/1662037560 learned_addr learned my addr 172.21.15.195:0/1662037560 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:13.126 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.122+0000 7f6abcbc1700 1 -- 172.21.15.195:0/1662037560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac005040 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6abcbc1700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6aac005010 tx=0x7f6aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aac009320 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab8060440 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6aac013380 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6aac014e50 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.123+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab8109b90 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.124+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6aac0263a0 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.124+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8045ad0 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.124+0000 7f6ab57fa700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 0x7f6aa80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.124+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6aac040f00 con 0x7f6ab80f5410 2024-02-20T00:29:13.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.124+0000 7f6ab7fff700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 0x7f6aa80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:13.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.125+0000 7f6ab7fff700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 0x7f6aa80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6aa40096a0 tx=0x7f6aa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:13.130 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.129+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6aac0126f0 con 0x7f6ab80f5410 2024-02-20T00:29:13.334 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.333+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ab810a3a0 con 0x7f6ab80f5410 2024-02-20T00:29:13.335 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.333+0000 7f6ab57fa700 1 -- 172.21.15.195:0/1662037560 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6aac012330 con 0x7f6ab80f5410 2024-02-20T00:29:13.335 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:13.335 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 msgr2=0x7f6aa80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 0x7f6aa80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6aa40096a0 tx=0x7f6aa4006b40).stop 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 msgr2=0x7f6ab81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6aac005010 tx=0x7f6aac00b040).stop 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 shutdown_connections 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6aa802ff10 0x7f6aa80323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 --2- 172.21.15.195:0/1662037560 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ab80f5410 0x7f6ab81094e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 >> 172.21.15.195:0/1662037560 conn(0x7f6ab80f0b70 msgr2=0x7f6ab80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 shutdown_connections 2024-02-20T00:29:13.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:13.335+0000 7f6abee25700 1 -- 172.21.15.195:0/1662037560 wait complete. 2024-02-20T00:29:13.338 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:12.555089+0000 mgr.smithi087.iydsqo (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:14.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:13.334447+0000 mon.smithi087 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.195:0/1662037560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:14.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:14.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:14.555341+0000 mgr.smithi087.iydsqo (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:16.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 -- 172.21.15.195:0/3902675012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 msgr2=0x7fce140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/3902675012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce140f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fce04009fb0 tx=0x7fce04009b30).stop 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 -- 172.21.15.195:0/3902675012 shutdown_connections 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/3902675012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce140f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 -- 172.21.15.195:0/3902675012 >> 172.21.15.195:0/3902675012 conn(0x7fce140f1c70 msgr2=0x7fce140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 -- 172.21.15.195:0/3902675012 shutdown_connections 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.713+0000 7fce1c4d6700 1 -- 172.21.15.195:0/3902675012 wait complete. 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1c4d6700 1 Processor -- start 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1c4d6700 1 -- start start 2024-02-20T00:29:16.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1c4d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1c4d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce14069e20 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1a272700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1a272700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60156/0 (socket says 172.21.15.195:60156) 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1a272700 1 -- 172.21.15.195:0/17762245 learned_addr learned my addr 172.21.15.195:0/17762245 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.714+0000 7fce1a272700 1 -- 172.21.15.195:0/17762245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04005040 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce1a272700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fce04005010 tx=0x7fce0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04009320 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce14069fc0 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce04013380 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04014e50 con 0x7fce140f6510 2024-02-20T00:29:16.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.715+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1406a450 con 0x7fce140f6510 2024-02-20T00:29:16.719 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce04014420 con 0x7fce140f6510 2024-02-20T00:29:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce14059100 con 0x7fce140f6510 2024-02-20T00:29:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce0affd700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 0x7fce000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce04040f90 con 0x7fce140f6510 2024-02-20T00:29:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce19a71700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 0x7fce000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:16.720 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.716+0000 7fce19a71700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 0x7fce000323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fce100096a0 tx=0x7fce10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:16.722 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.720+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce04014790 con 0x7fce140f6510 2024-02-20T00:29:16.925 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.924+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce140fc0a0 con 0x7fce140f6510 2024-02-20T00:29:16.926 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.924+0000 7fce0affd700 1 -- 172.21.15.195:0/17762245 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce0401d380 con 0x7fce140f6510 2024-02-20T00:29:16.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:16.926 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 msgr2=0x7fce000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 0x7fce000323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fce100096a0 tx=0x7fce10006b40).stop 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 msgr2=0x7fce1406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fce04005010 tx=0x7fce0400b040).stop 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 shutdown_connections 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fce0002ff10 0x7fce000323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 --2- 172.21.15.195:0/17762245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fce140f6510 0x7fce1406b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 >> 172.21.15.195:0/17762245 conn(0x7fce140f1c70 msgr2=0x7fce140f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 shutdown_connections 2024-02-20T00:29:16.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:16.926+0000 7fce1c4d6700 1 -- 172.21.15.195:0/17762245 wait complete. 2024-02-20T00:29:16.929 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:17.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:17 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:16.925521+0000 mon.smithi087 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.195:0/17762245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:18.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:18.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:16.555663+0000 mgr.smithi087.iydsqo (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:20.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 -- 172.21.15.195:0/3664779030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 msgr2=0x7f34e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3664779030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e00f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f34d4009fb0 tx=0x7f34d4009b30).stop 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 -- 172.21.15.195:0/3664779030 shutdown_connections 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3664779030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e00f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 -- 172.21.15.195:0/3664779030 >> 172.21.15.195:0/3664779030 conn(0x7f34e00f1c70 msgr2=0x7f34e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 -- 172.21.15.195:0/3664779030 shutdown_connections 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.273+0000 7f34e6d32700 1 -- 172.21.15.195:0/3664779030 wait complete. 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e6d32700 1 Processor -- start 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e6d32700 1 -- start start 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e6d32700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:20.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e6d32700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0106d00 con 0x7f34e00f6510 2024-02-20T00:29:20.278 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e4ace700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:20.278 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e4ace700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60176/0 (socket says 172.21.15.195:60176) 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e4ace700 1 -- 172.21.15.195:0/3633346406 learned_addr learned my addr 172.21.15.195:0/3633346406 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.274+0000 7f34e4ace700 1 -- 172.21.15.195:0/3633346406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d4005040 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34e4ace700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f34d4005010 tx=0x7f34d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34d4009320 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e0106f00 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34d4013380 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34d4014e50 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.275+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e0107390 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34d4014420 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34dd7fa700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 0x7f34d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34d4040e30 con 0x7f34e00f6510 2024-02-20T00:29:20.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e0059100 con 0x7f34e00f6510 2024-02-20T00:29:20.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34dffff700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 0x7f34d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:20.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.276+0000 7f34dffff700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 0x7f34d00323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f34cc0096a0 tx=0x7f34cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:20.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.281+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34d40095f0 con 0x7f34e00f6510 2024-02-20T00:29:20.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.486+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34e00fc0a0 con 0x7f34e00f6510 2024-02-20T00:29:20.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.487+0000 7f34dd7fa700 1 -- 172.21.15.195:0/3633346406 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34d402b020 con 0x7f34e00f6510 2024-02-20T00:29:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:20.488 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:20.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 msgr2=0x7f34d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:20.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 0x7f34d00323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f34cc0096a0 tx=0x7f34cc006b40).stop 2024-02-20T00:29:20.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 msgr2=0x7f34e0106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f34d4005010 tx=0x7f34d400b040).stop 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 shutdown_connections 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f34d002ff10 0x7f34d00323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 --2- 172.21.15.195:0/3633346406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34e00f6510 0x7f34e0106810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 >> 172.21.15.195:0/3633346406 conn(0x7f34e00f1c70 msgr2=0x7f34e00f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 shutdown_connections 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:20.488+0000 7f34e6d32700 1 -- 172.21.15.195:0/3633346406 wait complete. 2024-02-20T00:29:20.491 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:18.556022+0000 mgr.smithi087.iydsqo (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:21.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:21 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:20.488009+0000 mon.smithi087 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.195:0/3633346406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:22.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:22.150 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:20.556412+0000 mgr.smithi087.iydsqo (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:23.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 -- 172.21.15.195:0/383218511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 msgr2=0x7f26580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 --2- 172.21.15.195:0/383218511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f26580f90f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f264c009fb0 tx=0x7f264c009b30).stop 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 -- 172.21.15.195:0/383218511 shutdown_connections 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 --2- 172.21.15.195:0/383218511 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f26580f90f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 -- 172.21.15.195:0/383218511 >> 172.21.15.195:0/383218511 conn(0x7f26580f0b70 msgr2=0x7f26580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 -- 172.21.15.195:0/383218511 shutdown_connections 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.937+0000 7f265d2e3700 1 -- 172.21.15.195:0/383218511 wait complete. 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f265d2e3700 1 Processor -- start 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f265d2e3700 1 -- start start 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f265d2e3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:23.940 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f265d2e3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2658106c70 con 0x7f2658060280 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f2656ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f2656ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60200/0 (socket says 172.21.15.195:60200) 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f2656ffd700 1 -- 172.21.15.195:0/3548473424 learned_addr learned my addr 172.21.15.195:0/3548473424 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.938+0000 7f2656ffd700 1 -- 172.21.15.195:0/3548473424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c005040 con 0x7f2658060280 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f2656ffd700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f264c0089f0 tx=0x7f264c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:23.942 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f264c009320 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2658106e70 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f264c013380 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f264c014e20 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2658107300 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.939+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f264c0263a0 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.940+0000 7f263f7fe700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 0x7f26400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.940+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f264c040f90 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.940+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2658045ad0 con 0x7f2658060280 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.940+0000 7f26567fc700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 0x7f26400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:23.943 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.940+0000 7f26567fc700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 0x7f26400323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f26480096a0 tx=0x7f2648006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:23.946 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:23.944+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f264c012920 con 0x7f2658060280 2024-02-20T00:29:24.150 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.148+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2658107f80 con 0x7f2658060280 2024-02-20T00:29:24.150 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.149+0000 7f263f7fe700 1 -- 172.21.15.195:0/3548473424 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f264c02b020 con 0x7f2658060280 2024-02-20T00:29:24.151 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:24.151 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 msgr2=0x7f26400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 0x7f26400323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f26480096a0 tx=0x7f2648006b40).stop 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 msgr2=0x7f2658106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f264c0089f0 tx=0x7f264c00b040).stop 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 shutdown_connections 2024-02-20T00:29:24.152 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f264002ff10 0x7f26400323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:24.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 --2- 172.21.15.195:0/3548473424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2658060280 0x7f2658106780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:24.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 >> 172.21.15.195:0/3548473424 conn(0x7f26580f0b70 msgr2=0x7f26580f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:24.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 shutdown_connections 2024-02-20T00:29:24.153 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:24.150+0000 7f265d2e3700 1 -- 172.21.15.195:0/3548473424 wait complete. 2024-02-20T00:29:24.153 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:24.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:22.556746+0000 mgr.smithi087.iydsqo (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:24.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:24.149858+0000 mon.smithi087 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.195:0/3548473424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:25.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:25.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:26.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:24.557077+0000 mgr.smithi087.iydsqo (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:27.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 -- 172.21.15.195:0/1565795663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 msgr2=0x7fe86c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:27.606 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1565795663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c0f90f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe860009fb0 tx=0x7fe860009b30).stop 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 -- 172.21.15.195:0/1565795663 shutdown_connections 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1565795663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c0f90f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 -- 172.21.15.195:0/1565795663 >> 172.21.15.195:0/1565795663 conn(0x7fe86c0f0b70 msgr2=0x7fe86c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 -- 172.21.15.195:0/1565795663 shutdown_connections 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 -- 172.21.15.195:0/1565795663 wait complete. 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.605+0000 7fe872a2a700 1 Processor -- start 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe872a2a700 1 -- start start 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe872a2a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe872a2a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c106c70 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe86bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe86bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49382/0 (socket says 172.21.15.195:49382) 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe86bfff700 1 -- 172.21.15.195:0/1834436709 learned_addr learned my addr 172.21.15.195:0/1834436709 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe86bfff700 1 -- 172.21.15.195:0/1834436709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860005040 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.606+0000 7fe86bfff700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe8600089f0 tx=0x7fe86000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe860009320 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe860013380 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c106e70 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe860014e20 con 0x7fe86c060280 2024-02-20T00:29:27.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c107300 con 0x7fe86c060280 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.607+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8600263a0 con 0x7fe86c060280 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.608+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c045ad0 con 0x7fe86c060280 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.608+0000 7fe868ff9700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 0x7fe8540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.608+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe860041230 con 0x7fe86c060280 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.608+0000 7fe86b7fe700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 0x7fe8540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:27.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.608+0000 7fe86b7fe700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 0x7fe8540323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe85c0096a0 tx=0x7fe85c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:27.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.612+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe86001ccf0 con 0x7fe86c060280 2024-02-20T00:29:27.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.816+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe86c1081b0 con 0x7fe86c060280 2024-02-20T00:29:27.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.817+0000 7fe868ff9700 1 -- 172.21.15.195:0/1834436709 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe86001ccf0 con 0x7fe86c060280 2024-02-20T00:29:27.818 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:27.818 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:27.819 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 msgr2=0x7fe8540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 0x7fe8540323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe85c0096a0 tx=0x7fe85c006b40).stop 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 msgr2=0x7fe86c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe8600089f0 tx=0x7fe86000b040).stop 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 shutdown_connections 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe85402ff10 0x7fe8540323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 --2- 172.21.15.195:0/1834436709 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe86c060280 0x7fe86c106780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 >> 172.21.15.195:0/1834436709 conn(0x7fe86c0f0b70 msgr2=0x7fe86c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 shutdown_connections 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:27.819+0000 7fe872a2a700 1 -- 172.21.15.195:0/1834436709 wait complete. 2024-02-20T00:29:27.820 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:28.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:26.557426+0000 mgr.smithi087.iydsqo (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:28.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:27.817446+0000 mon.smithi087 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.195:0/1834436709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:29.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:29.476 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:28.557796+0000 mgr.smithi087.iydsqo (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- 172.21.15.195:0/4066495120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 msgr2=0x7f62d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 --2- 172.21.15.195:0/4066495120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d00f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f62c0009fb0 tx=0x7f62c0009b30).stop 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- 172.21.15.195:0/4066495120 shutdown_connections 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 --2- 172.21.15.195:0/4066495120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d00f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- 172.21.15.195:0/4066495120 >> 172.21.15.195:0/4066495120 conn(0x7f62d00f09a0 msgr2=0x7f62d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- 172.21.15.195:0/4066495120 shutdown_connections 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- 172.21.15.195:0/4066495120 wait complete. 2024-02-20T00:29:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 Processor -- start 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.349+0000 7f62d57ed700 1 -- start start 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62d57ed700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62d57ed700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0105990 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62cffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62cffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49404/0 (socket says 172.21.15.195:49404) 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62cffff700 1 -- 172.21.15.195:0/2186246759 learned_addr learned my addr 172.21.15.195:0/2186246759 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62cffff700 1 -- 172.21.15.195:0/2186246759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0005040 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62cffff700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f62c00089f0 tx=0x7f62c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.350+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c0009320 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d0105b90 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62c0013380 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c0014e20 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d0106020 con 0x7f62d00f51a0 2024-02-20T00:29:31.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62c0014420 con 0x7f62d00f51a0 2024-02-20T00:29:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62ccff9700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 0x7f62b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.351+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0045ad0 con 0x7f62d00f51a0 2024-02-20T00:29:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.352+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62c0041010 con 0x7f62d00f51a0 2024-02-20T00:29:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.352+0000 7f62cf7fe700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 0x7f62b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:31.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.352+0000 7f62cf7fe700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 0x7f62b80323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f62c40096a0 tx=0x7f62c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:31.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.356+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62c00095f0 con 0x7f62d00f51a0 2024-02-20T00:29:31.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.560+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62d0106ed0 con 0x7f62d00f51a0 2024-02-20T00:29:31.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.560+0000 7f62ccff9700 1 -- 172.21.15.195:0/2186246759 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62c001db70 con 0x7f62d00f51a0 2024-02-20T00:29:31.562 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:31.563 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 msgr2=0x7f62b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 0x7f62b80323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f62c40096a0 tx=0x7f62c4006b40).stop 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 msgr2=0x7f62d01054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f62c00089f0 tx=0x7f62c000b040).stop 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 shutdown_connections 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f62b802ff10 0x7f62b80323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 --2- 172.21.15.195:0/2186246759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62d00f51a0 0x7f62d01054a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 >> 172.21.15.195:0/2186246759 conn(0x7f62d00f09a0 msgr2=0x7f62d00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 shutdown_connections 2024-02-20T00:29:31.564 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:31.562+0000 7f62d57ed700 1 -- 172.21.15.195:0/2186246759 wait complete. 2024-02-20T00:29:31.565 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:32.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:30.558145+0000 mgr.smithi087.iydsqo (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:32.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:31.561625+0000 mon.smithi087 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.195:0/2186246759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:33.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:33.234 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:34.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:32.558506+0000 mgr.smithi087.iydsqo (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:35.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.102+0000 7fd9dd27a700 1 -- 172.21.15.195:0/3012385538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 msgr2=0x7fd9d80f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:35.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.102+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/3012385538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d80f5bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd9c8009fb0 tx=0x7fd9c8009b30).stop 2024-02-20T00:29:35.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- 172.21.15.195:0/3012385538 shutdown_connections 2024-02-20T00:29:35.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/3012385538 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d80f5bb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:35.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- 172.21.15.195:0/3012385538 >> 172.21.15.195:0/3012385538 conn(0x7fd9d80f0120 msgr2=0x7fd9d80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:35.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- 172.21.15.195:0/3012385538 shutdown_connections 2024-02-20T00:29:35.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- 172.21.15.195:0/3012385538 wait complete. 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 Processor -- start 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- start start 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.103+0000 7fd9dd27a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80f5170 con 0x7fd9d80f5790 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49410/0 (socket says 172.21.15.195:49410) 2024-02-20T00:29:35.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d7fff700 1 -- 172.21.15.195:0/1084345822 learned_addr learned my addr 172.21.15.195:0/1084345822 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d7fff700 1 -- 172.21.15.195:0/1084345822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c8005040 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d7fff700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd9c8005010 tx=0x7fd9c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c8009320 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d80f5370 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9c8013380 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c8014e50 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.104+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d8109990 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.105+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9c80263a0 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.105+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d8045ad0 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.105+0000 7fd9d4ff9700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.105+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9c8041280 con 0x7fd9d80f5790 2024-02-20T00:29:35.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.106+0000 7fd9d77fe700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:35.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.106+0000 7fd9d77fe700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd9cc0096a0 tx=0x7fd9cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:35.111 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.110+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9c8012330 con 0x7fd9d80f5790 2024-02-20T00:29:35.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.312+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9d810a1a0 con 0x7fd9d80f5790 2024-02-20T00:29:35.315 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.313+0000 7fd9d4ff9700 1 -- 172.21.15.195:0/1084345822 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9c8012330 con 0x7fd9d80f5790 2024-02-20T00:29:35.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:35.315 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:35.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.314+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 msgr2=0x7fd9c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:35.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.314+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd9cc0096a0 tx=0x7fd9cc006b40).stop 2024-02-20T00:29:35.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.314+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 msgr2=0x7fd9d81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:35.316 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.314+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd9c8005010 tx=0x7fd9c800b040).stop 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 shutdown_connections 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd9c002ff10 0x7fd9c00323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 --2- 172.21.15.195:0/1084345822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80f5790 0x7fd9d81092e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 >> 172.21.15.195:0/1084345822 conn(0x7fd9d80f0120 msgr2=0x7fd9d80f0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 shutdown_connections 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:35.315+0000 7fd9dd27a700 1 -- 172.21.15.195:0/1084345822 wait complete. 2024-02-20T00:29:35.317 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:36.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:34.558765+0000 mgr.smithi087.iydsqo (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:36.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:35.314295+0000 mon.smithi087 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.195:0/1084345822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:36.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:36.924 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 -- 172.21.15.195:0/2138016063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 msgr2=0x7f6b780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 --2- 172.21.15.195:0/2138016063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b780f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6b6c009fb0 tx=0x7f6b6c009b30).stop 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 -- 172.21.15.195:0/2138016063 shutdown_connections 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 --2- 172.21.15.195:0/2138016063 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b780f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 -- 172.21.15.195:0/2138016063 >> 172.21.15.195:0/2138016063 conn(0x7f6b780f0b70 msgr2=0x7f6b780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 -- 172.21.15.195:0/2138016063 shutdown_connections 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.676+0000 7f6b7f636700 1 -- 172.21.15.195:0/2138016063 wait complete. 2024-02-20T00:29:38.680 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7f636700 1 Processor -- start 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7f636700 1 -- start start 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7f636700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7f636700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b78105bd0 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7d3d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7d3d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38572/0 (socket says 172.21.15.195:38572) 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.677+0000 7f6b7d3d2700 1 -- 172.21.15.195:0/1733430818 learned_addr learned my addr 172.21.15.195:0/1733430818 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b7d3d2700 1 -- 172.21.15.195:0/1733430818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c005040 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b7d3d2700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6b6c005010 tx=0x7f6b6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b6c009320 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b6c013380 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b78105dd0 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b6c014e50 con 0x7f6b780f5410 2024-02-20T00:29:38.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.678+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b78106260 con 0x7f6b780f5410 2024-02-20T00:29:38.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.679+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b6c0263a0 con 0x7f6b780f5410 2024-02-20T00:29:38.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.679+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b78045ad0 con 0x7f6b780f5410 2024-02-20T00:29:38.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.679+0000 7f6b69ffb700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 0x7f6b640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:38.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.679+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b6c040f50 con 0x7f6b780f5410 2024-02-20T00:29:38.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.679+0000 7f6b7cbd1700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 0x7f6b640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:38.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.680+0000 7f6b7cbd1700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 0x7f6b640323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f6b740096a0 tx=0x7f6b74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:38.685 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.684+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b6c012330 con 0x7f6b780f5410 2024-02-20T00:29:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:36.559116+0000 mgr.smithi087.iydsqo (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:38.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.887+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b78107110 con 0x7f6b780f5410 2024-02-20T00:29:38.889 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.888+0000 7f6b69ffb700 1 -- 172.21.15.195:0/1733430818 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b6c01db70 con 0x7f6b780f5410 2024-02-20T00:29:38.889 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:38.890 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:38.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 msgr2=0x7f6b640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:38.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 0x7f6b640323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f6b740096a0 tx=0x7f6b74006b40).stop 2024-02-20T00:29:38.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 msgr2=0x7f6b781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:38.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6b6c005010 tx=0x7f6b6c00b040).stop 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 shutdown_connections 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6b6402ff10 0x7f6b640323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 --2- 172.21.15.195:0/1733430818 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6b780f5410 0x7f6b781056e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.889+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 >> 172.21.15.195:0/1733430818 conn(0x7f6b780f0b70 msgr2=0x7f6b780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.890+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 shutdown_connections 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:38.890+0000 7f6b7f636700 1 -- 172.21.15.195:0/1733430818 wait complete. 2024-02-20T00:29:38.892 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:39.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:39 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:38.889124+0000 mon.smithi087 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.195:0/1733430818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:40.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:40.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:38.559463+0000 mgr.smithi087.iydsqo (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:42.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 -- 172.21.15.195:0/2945956829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 msgr2=0x7fe6d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/2945956829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d40f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe6c8009fb0 tx=0x7fe6c8009b30).stop 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 -- 172.21.15.195:0/2945956829 shutdown_connections 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/2945956829 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d40f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 -- 172.21.15.195:0/2945956829 >> 172.21.15.195:0/2945956829 conn(0x7fe6d40f1c70 msgr2=0x7fe6d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 -- 172.21.15.195:0/2945956829 shutdown_connections 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 -- 172.21.15.195:0/2945956829 wait complete. 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.216+0000 7fe6d8e43700 1 Processor -- start 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d8e43700 1 -- start start 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d8e43700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d8e43700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d4106cd0 con 0x7fe6d40f6510 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:42.220 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38588/0 (socket says 172.21.15.195:38588) 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d259c700 1 -- 172.21.15.195:0/3612899527 learned_addr learned my addr 172.21.15.195:0/3612899527 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d259c700 1 -- 172.21.15.195:0/3612899527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8005040 con 0x7fe6d40f6510 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.217+0000 7fe6d259c700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe6c8005010 tx=0x7fe6c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6c8009320 con 0x7fe6d40f6510 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6c8013380 con 0x7fe6d40f6510 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d4106ed0 con 0x7fe6d40f6510 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6c8014e50 con 0x7fe6d40f6510 2024-02-20T00:29:42.221 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d4107360 con 0x7fe6d40f6510 2024-02-20T00:29:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.218+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6c8014420 con 0x7fe6d40f6510 2024-02-20T00:29:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.219+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d4045ad0 con 0x7fe6d40f6510 2024-02-20T00:29:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.219+0000 7fe6c2ffd700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 0x7fe6bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.219+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6c8040ee0 con 0x7fe6d40f6510 2024-02-20T00:29:42.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.219+0000 7fe6d1d9b700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 0x7fe6bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:42.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.219+0000 7fe6d1d9b700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 0x7fe6bc0323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe6c40096a0 tx=0x7fe6c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:42.225 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.223+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6c80095f0 con 0x7fe6d40f6510 2024-02-20T00:29:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.431+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6d4108210 con 0x7fe6d40f6510 2024-02-20T00:29:42.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.432+0000 7fe6c2ffd700 1 -- 172.21.15.195:0/3612899527 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6c801db70 con 0x7fe6d40f6510 2024-02-20T00:29:42.434 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:42.434 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:42.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 msgr2=0x7fe6bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:42.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 0x7fe6bc0323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fe6c40096a0 tx=0x7fe6c4006b40).stop 2024-02-20T00:29:42.435 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 msgr2=0x7fe6d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe6c8005010 tx=0x7fe6c800b040).stop 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 shutdown_connections 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6bc02ff10 0x7fe6bc0323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 --2- 172.21.15.195:0/3612899527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6d40f6510 0x7fe6d41067e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.433+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 >> 172.21.15.195:0/3612899527 conn(0x7fe6d40f1c70 msgr2=0x7fe6d40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.434+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 shutdown_connections 2024-02-20T00:29:42.436 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:42.434+0000 7fe6d8e43700 1 -- 172.21.15.195:0/3612899527 wait complete. 2024-02-20T00:29:42.437 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:42.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:40.559818+0000 mgr.smithi087.iydsqo (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:43.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:43 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:42.432920+0000 mon.smithi087 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.195:0/3612899527' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:44.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:44.041 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:44.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:42.560070+0000 mgr.smithi087.iydsqo (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 -- 172.21.15.195:0/2409078032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 msgr2=0x7efe980f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/2409078032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe980f90f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efe88009fb0 tx=0x7efe88009b30).stop 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 -- 172.21.15.195:0/2409078032 shutdown_connections 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/2409078032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe980f90f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 -- 172.21.15.195:0/2409078032 >> 172.21.15.195:0/2409078032 conn(0x7efe980f0b70 msgr2=0x7efe980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 -- 172.21.15.195:0/2409078032 shutdown_connections 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.993+0000 7efe9e9f5700 1 -- 172.21.15.195:0/2409078032 wait complete. 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe9e9f5700 1 Processor -- start 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe9e9f5700 1 -- start start 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe9e9f5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe9e9f5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98106c70 con 0x7efe98060280 2024-02-20T00:29:45.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe97fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe97fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38596/0 (socket says 172.21.15.195:38596) 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe97fff700 1 -- 172.21.15.195:0/490177606 learned_addr learned my addr 172.21.15.195:0/490177606 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe97fff700 1 -- 172.21.15.195:0/490177606 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88005040 con 0x7efe98060280 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.994+0000 7efe97fff700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7efe880089f0 tx=0x7efe8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe88009320 con 0x7efe98060280 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe98106e70 con 0x7efe98060280 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe88013380 con 0x7efe98060280 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe88014e20 con 0x7efe98060280 2024-02-20T00:29:45.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe98107300 con 0x7efe98060280 2024-02-20T00:29:45.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.995+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe88014420 con 0x7efe98060280 2024-02-20T00:29:45.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.996+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98045ad0 con 0x7efe98060280 2024-02-20T00:29:45.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.996+0000 7efe94ff9700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 0x7efe800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:45.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.996+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe88040f60 con 0x7efe98060280 2024-02-20T00:29:45.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.996+0000 7efe977fe700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 0x7efe800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:46.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:45.996+0000 7efe977fe700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 0x7efe800323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efe8c0096a0 tx=0x7efe8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:46.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.001+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe8801d980 con 0x7efe98060280 2024-02-20T00:29:46.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.204+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe981081b0 con 0x7efe98060280 2024-02-20T00:29:46.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.205+0000 7efe94ff9700 1 -- 172.21.15.195:0/490177606 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe88010230 con 0x7efe98060280 2024-02-20T00:29:46.207 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:46.207 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 msgr2=0x7efe800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 0x7efe800323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efe8c0096a0 tx=0x7efe8c006b40).stop 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 msgr2=0x7efe98106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7efe880089f0 tx=0x7efe8800b040).stop 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 shutdown_connections 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efe8002ff10 0x7efe800323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 --2- 172.21.15.195:0/490177606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efe98060280 0x7efe98106780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 >> 172.21.15.195:0/490177606 conn(0x7efe980f0b70 msgr2=0x7efe980f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 shutdown_connections 2024-02-20T00:29:46.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:46.207+0000 7efe9e9f5700 1 -- 172.21.15.195:0/490177606 wait complete. 2024-02-20T00:29:46.210 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:46.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:44.560368+0000 mgr.smithi087.iydsqo (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:46.206486+0000 mon.smithi087 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.195:0/490177606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:47.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:47.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:46.560738+0000 mgr.smithi087.iydsqo (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:49.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 -- 172.21.15.195:0/419377790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 msgr2=0x7f11740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 --2- 172.21.15.195:0/419377790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11740f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1168009fb0 tx=0x7f1168009b30).stop 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 -- 172.21.15.195:0/419377790 shutdown_connections 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 --2- 172.21.15.195:0/419377790 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11740f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 -- 172.21.15.195:0/419377790 >> 172.21.15.195:0/419377790 conn(0x7f11740f0b70 msgr2=0x7f11740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 -- 172.21.15.195:0/419377790 shutdown_connections 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.664+0000 7f11788c2700 1 -- 172.21.15.195:0/419377790 wait complete. 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f11788c2700 1 Processor -- start 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f11788c2700 1 -- start start 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f11788c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:49.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f11788c2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174106cc0 con 0x7f11740f5410 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f117259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f117259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59018/0 (socket says 172.21.15.195:59018) 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f117259c700 1 -- 172.21.15.195:0/3914288115 learned_addr learned my addr 172.21.15.195:0/3914288115 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.665+0000 7f117259c700 1 -- 172.21.15.195:0/3914288115 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168005040 con 0x7f11740f5410 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f117259c700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1168005010 tx=0x7f116800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1168009320 con 0x7f11740f5410 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1168013380 con 0x7f11740f5410 2024-02-20T00:29:49.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1174106ec0 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1168014e50 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.666+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1174107350 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11680095f0 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174045ad0 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f1162ffd700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 0x7f115c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1168040f60 con 0x7f11740f5410 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f1171d9b700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 0x7f115c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:49.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.667+0000 7f1171d9b700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 0x7f115c0323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f11640096a0 tx=0x7f1164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:49.673 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.671+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1168014420 con 0x7f11740f5410 2024-02-20T00:29:49.878 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.877+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1174108200 con 0x7f11740f5410 2024-02-20T00:29:49.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.877+0000 7f1162ffd700 1 -- 172.21.15.195:0/3914288115 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f116801d980 con 0x7f11740f5410 2024-02-20T00:29:49.879 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:49.880 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:49.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 msgr2=0x7f115c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:49.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 0x7f115c0323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f11640096a0 tx=0x7f1164006b40).stop 2024-02-20T00:29:49.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 msgr2=0x7f11741067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:49.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1168005010 tx=0x7f116800b040).stop 2024-02-20T00:29:49.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 shutdown_connections 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f115c02ff10 0x7f115c0323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 --2- 172.21.15.195:0/3914288115 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11740f5410 0x7f11741067d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 >> 172.21.15.195:0/3914288115 conn(0x7f11740f0b70 msgr2=0x7f117405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 shutdown_connections 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:49.879+0000 7f11788c2700 1 -- 172.21.15.195:0/3914288115 wait complete. 2024-02-20T00:29:49.882 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:50.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:48.561024+0000 mgr.smithi087.iydsqo (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:50.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:49.878816+0000 mon.smithi087 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.195:0/3914288115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:51.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:51.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:50.561339+0000 mgr.smithi087.iydsqo (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.344+0000 7f93b4a77700 1 -- 172.21.15.195:0/3172168990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 msgr2=0x7f93b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.344+0000 7f93b4a77700 1 --2- 172.21.15.195:0/3172168990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b00f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f9398009fb0 tx=0x7f9398009b30).stop 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- 172.21.15.195:0/3172168990 shutdown_connections 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 --2- 172.21.15.195:0/3172168990 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b00f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- 172.21.15.195:0/3172168990 >> 172.21.15.195:0/3172168990 conn(0x7f93b00f0b70 msgr2=0x7f93b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- 172.21.15.195:0/3172168990 shutdown_connections 2024-02-20T00:29:53.347 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- 172.21.15.195:0/3172168990 wait complete. 2024-02-20T00:29:53.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 Processor -- start 2024-02-20T00:29:53.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- start start 2024-02-20T00:29:53.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:53.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.345+0000 7f93b4a77700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b0105c10 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93ae59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93ae59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59028/0 (socket says 172.21.15.195:59028) 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93ae59c700 1 -- 172.21.15.195:0/2603633232 learned_addr learned my addr 172.21.15.195:0/2603633232 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93ae59c700 1 -- 172.21.15.195:0/2603633232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398005040 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93ae59c700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f9398005010 tx=0x7f939800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9398009320 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b0105e10 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9398013380 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9398014e50 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.346+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01062a0 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.347+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93980095f0 con 0x7f93b00f5410 2024-02-20T00:29:53.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.347+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b0045ad0 con 0x7f93b00f5410 2024-02-20T00:29:53.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.347+0000 7f93a6ffd700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 0x7f939c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:53.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.348+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9398040e90 con 0x7f93b00f5410 2024-02-20T00:29:53.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.348+0000 7f93add9b700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 0x7f939c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:53.351 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.348+0000 7f93add9b700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 0x7f939c0323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f93a00096a0 tx=0x7f93a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:53.354 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.352+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9398014420 con 0x7f93b00f5410 2024-02-20T00:29:53.560 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.558+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93b0107150 con 0x7f93b00f5410 2024-02-20T00:29:53.560 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.559+0000 7f93a6ffd700 1 -- 172.21.15.195:0/2603633232 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9398010230 con 0x7f93b00f5410 2024-02-20T00:29:53.561 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:53.561 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:53.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 msgr2=0x7f939c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:53.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 0x7f939c0323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f93a00096a0 tx=0x7f93a0006b40).stop 2024-02-20T00:29:53.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 msgr2=0x7f93b0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f9398005010 tx=0x7f939800b040).stop 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 shutdown_connections 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f939c02ff10 0x7f939c0323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 --2- 172.21.15.195:0/2603633232 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93b00f5410 0x7f93b0105720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 >> 172.21.15.195:0/2603633232 conn(0x7f93b00f0b70 msgr2=0x7f93b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 shutdown_connections 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:53.560+0000 7f93b4a77700 1 -- 172.21.15.195:0/2603633232 wait complete. 2024-02-20T00:29:53.563 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:54.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:52.561688+0000 mgr.smithi087.iydsqo (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:54.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:53.560043+0000 mon.smithi087 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.195:0/2603633232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:55.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:55.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:29:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:54.562054+0000 mgr.smithi087.iydsqo (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:57.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 -- 172.21.15.195:0/4253038406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 msgr2=0x7fd0c00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:57.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/4253038406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c00f90f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd0b4009fb0 tx=0x7fd0b4009b30).stop 2024-02-20T00:29:57.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 -- 172.21.15.195:0/4253038406 shutdown_connections 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/4253038406 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c00f90f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 -- 172.21.15.195:0/4253038406 >> 172.21.15.195:0/4253038406 conn(0x7fd0c00f0b70 msgr2=0x7fd0c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 -- 172.21.15.195:0/4253038406 shutdown_connections 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 -- 172.21.15.195:0/4253038406 wait complete. 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.135+0000 7fd0c5be4700 1 Processor -- start 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0c5be4700 1 -- start start 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0c5be4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:57.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0c5be4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0106c70 con 0x7fd0c0060280 2024-02-20T00:29:57.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:57.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0bf7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:43834/0 (socket says 172.21.15.195:43834) 2024-02-20T00:29:57.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0bf7fe700 1 -- 172.21.15.195:0/3783483931 learned_addr learned my addr 172.21.15.195:0/3783483931 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:29:57.140 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0bf7fe700 1 -- 172.21.15.195:0/3783483931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b4005040 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.136+0000 7fd0bf7fe700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd0b40089f0 tx=0x7fd0b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.137+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0b4009320 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.137+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c0106e70 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.137+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0b4013380 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.137+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0b4014e20 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.137+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c0107300 con 0x7fd0c0060280 2024-02-20T00:29:57.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0b4014420 con 0x7fd0c0060280 2024-02-20T00:29:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c0045ad0 con 0x7fd0c0060280 2024-02-20T00:29:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0a7fff700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 0x7fd0a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:29:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0b4040f20 con 0x7fd0c0060280 2024-02-20T00:29:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0beffd700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 0x7fd0a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:29:57.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.138+0000 7fd0beffd700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 0x7fd0a80323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:29:57.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.142+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0b406d2b0 con 0x7fd0c0060280 2024-02-20T00:29:57.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.348+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0c01081b0 con 0x7fd0c0060280 2024-02-20T00:29:57.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.349+0000 7fd0a7fff700 1 -- 172.21.15.195:0/3783483931 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0b40095f0 con 0x7fd0c0060280 2024-02-20T00:29:57.351 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:29:57.351 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:29:57.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.350+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 msgr2=0x7fd0a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:57.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.350+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 0x7fd0a80323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).stop 2024-02-20T00:29:57.352 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 msgr2=0x7fd0c0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd0b40089f0 tx=0x7fd0b400b040).stop 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 shutdown_connections 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd0a802ff10 0x7fd0a80323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 --2- 172.21.15.195:0/3783483931 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd0c0060280 0x7fd0c0106780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 >> 172.21.15.195:0/3783483931 conn(0x7fd0c00f0b70 msgr2=0x7fd0c00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 shutdown_connections 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:29:57.351+0000 7fd0c5be4700 1 -- 172.21.15.195:0/3783483931 wait complete. 2024-02-20T00:29:57.353 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:29:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:56.667375+0000 mon.smithi087 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:29:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:56.668753+0000 mon.smithi087 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:29:58.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:56.562352+0000 mgr.smithi087.iydsqo (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:29:58.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:29:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:29:57.349856+0000 mon.smithi087 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.195:0/3783483931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:29:59.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:29:59.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:29:58.562711+0000 mgr.smithi087.iydsqo (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 -- 172.21.15.195:0/1191068773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 msgr2=0x7f60600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 --2- 172.21.15.195:0/1191068773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f60600f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f6054009fb0 tx=0x7f6054009b30).stop 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 -- 172.21.15.195:0/1191068773 shutdown_connections 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 --2- 172.21.15.195:0/1191068773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f60600f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 -- 172.21.15.195:0/1191068773 >> 172.21.15.195:0/1191068773 conn(0x7f60600f0b70 msgr2=0x7f60600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 -- 172.21.15.195:0/1191068773 shutdown_connections 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.835+0000 7f6066af9700 1 -- 172.21.15.195:0/1191068773 wait complete. 2024-02-20T00:30:00.879 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6066af9700 1 Processor -- start 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6066af9700 1 -- start start 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6066af9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6066af9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6060060240 con 0x7f60600f5410 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6064895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6064895700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:43848/0 (socket says 172.21.15.195:43848) 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6064895700 1 -- 172.21.15.195:0/3104135997 learned_addr learned my addr 172.21.15.195:0/3104135997 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6064895700 1 -- 172.21.15.195:0/3104135997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6054005040 con 0x7f60600f5410 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.836+0000 7f6064895700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6054005010 tx=0x7f605400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054009320 con 0x7f60600f5410 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6060060440 con 0x7f60600f5410 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6054013380 con 0x7f60600f5410 2024-02-20T00:30:00.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6054014e50 con 0x7f60600f5410 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6060109bc0 con 0x7f60600f5410 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6054014420 con 0x7f60600f5410 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.837+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6060045ad0 con 0x7f60600f5410 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.838+0000 7f605d7fa700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 0x7f60500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.838+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6054040f40 con 0x7f60600f5410 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.838+0000 7f605ffff700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 0x7f60500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.838+0000 7f605ffff700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 0x7f60500323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f604c0096a0 tx=0x7f604c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:00.881 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:00.842+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60540095f0 con 0x7f60600f5410 2024-02-20T00:30:01.047 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.045+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f606010a3e0 con 0x7f60600f5410 2024-02-20T00:30:01.048 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.047+0000 7f605d7fa700 1 -- 172.21.15.195:0/3104135997 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f605401db70 con 0x7f60600f5410 2024-02-20T00:30:01.048 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:01.048 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:01.049 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 msgr2=0x7f60500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 0x7f60500323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f604c0096a0 tx=0x7f604c006b40).stop 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 msgr2=0x7f6060109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6054005010 tx=0x7f605400b040).stop 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 shutdown_connections 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f605002ff10 0x7f60500323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 --2- 172.21.15.195:0/3104135997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60600f5410 0x7f6060109510 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 >> 172.21.15.195:0/3104135997 conn(0x7f60600f0b70 msgr2=0x7f60600f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 shutdown_connections 2024-02-20T00:30:01.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:01.048+0000 7f6066af9700 1 -- 172.21.15.195:0/3104135997 wait complete. 2024-02-20T00:30:01.051 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:01.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:01 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:01.047223+0000 mon.smithi087 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.195:0/3104135997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:02.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:02.736 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:02.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:00.563081+0000 mgr.smithi087.iydsqo (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 -- 172.21.15.195:0/853679899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 msgr2=0x7ff7600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 --2- 172.21.15.195:0/853679899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7600f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff748009fb0 tx=0x7ff748009b30).stop 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 -- 172.21.15.195:0/853679899 shutdown_connections 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 --2- 172.21.15.195:0/853679899 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7600f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 -- 172.21.15.195:0/853679899 >> 172.21.15.195:0/853679899 conn(0x7ff7600f1c70 msgr2=0x7ff7600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 -- 172.21.15.195:0/853679899 shutdown_connections 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.505+0000 7ff764cbd700 1 -- 172.21.15.195:0/853679899 wait complete. 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff764cbd700 1 Processor -- start 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff764cbd700 1 -- start start 2024-02-20T00:30:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff764cbd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff764cbd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760106cd0 con 0x7ff7600f6510 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff75e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff75e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:43864/0 (socket says 172.21.15.195:43864) 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff75e59c700 1 -- 172.21.15.195:0/2914761975 learned_addr learned my addr 172.21.15.195:0/2914761975 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.506+0000 7ff75e59c700 1 -- 172.21.15.195:0/2914761975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748005040 con 0x7ff7600f6510 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff75e59c700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7ff7480089f0 tx=0x7ff74800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff748009320 con 0x7ff7600f6510 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff748013380 con 0x7ff7600f6510 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74801ed80 con 0x7ff7600f6510 2024-02-20T00:30:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff760106ed0 con 0x7ff7600f6510 2024-02-20T00:30:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.507+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff760107360 con 0x7ff7600f6510 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff74801e420 con 0x7ff7600f6510 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760045ad0 con 0x7ff7600f6510 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff756ffd700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 0x7ff74c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff748040f80 con 0x7ff7600f6510 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff75dd9b700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 0x7ff74c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.508+0000 7ff75dd9b700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 0x7ff74c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff7500096a0 tx=0x7ff750006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.513+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74806d2b0 con 0x7ff7600f6510 2024-02-20T00:30:04.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.719+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff760108210 con 0x7ff7600f6510 2024-02-20T00:30:04.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.720+0000 7ff756ffd700 1 -- 172.21.15.195:0/2914761975 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7480095f0 con 0x7ff7600f6510 2024-02-20T00:30:04.722 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:04.722 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 msgr2=0x7ff74c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 0x7ff74c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff7500096a0 tx=0x7ff750006b40).stop 2024-02-20T00:30:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 msgr2=0x7ff7601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7ff7480089f0 tx=0x7ff74800b040).stop 2024-02-20T00:30:04.723 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 shutdown_connections 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff74c02ff60 0x7ff74c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 --2- 172.21.15.195:0/2914761975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7600f6510 0x7ff7601067e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.721+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 >> 172.21.15.195:0/2914761975 conn(0x7ff7600f1c70 msgr2=0x7ff7600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.722+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 shutdown_connections 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:04.722+0000 7ff764cbd700 1 -- 172.21.15.195:0/2914761975 wait complete. 2024-02-20T00:30:04.724 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:04.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:02.563411+0000 mgr.smithi087.iydsqo (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:05.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:05 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:04.721132+0000 mon.smithi087 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.195:0/2914761975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:06.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:06.318 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:06.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:04.563706+0000 mgr.smithi087.iydsqo (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:08.261 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 -- 172.21.15.195:0/2372584327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 msgr2=0x7f75480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 --2- 172.21.15.195:0/2372584327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f8f20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f753c009fb0 tx=0x7f753c009b30).stop 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 -- 172.21.15.195:0/2372584327 shutdown_connections 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 --2- 172.21.15.195:0/2372584327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f8f20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 -- 172.21.15.195:0/2372584327 >> 172.21.15.195:0/2372584327 conn(0x7f75480f09a0 msgr2=0x7f75480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 -- 172.21.15.195:0/2372584327 shutdown_connections 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.258+0000 7f754759e700 1 -- 172.21.15.195:0/2372584327 wait complete. 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754759e700 1 Processor -- start 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754759e700 1 -- start start 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754759e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75480f4e20 con 0x7f75480601e0 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:08.262 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57894/0 (socket says 172.21.15.195:57894) 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754659c700 1 -- 172.21.15.195:0/3373493036 learned_addr learned my addr 172.21.15.195:0/3373493036 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754659c700 1 -- 172.21.15.195:0/3373493036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c005040 con 0x7f75480601e0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.259+0000 7f754659c700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f753c005010 tx=0x7f753c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.260+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c009320 con 0x7f75480601e0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.260+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75480f4fc0 con 0x7f75480601e0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.260+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f753c013380 con 0x7f75480601e0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.260+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c014e50 con 0x7f75480601e0 2024-02-20T00:30:08.263 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.260+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75480f5450 con 0x7f75480601e0 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f753c0095f0 con 0x7f75480601e0 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548059100 con 0x7f75480601e0 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f7536ffd700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 0x7f75300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f753c040e70 con 0x7f75480601e0 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f7545d9b700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 0x7f75300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:08.264 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.261+0000 7f7545d9b700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 0x7f75300323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f75380096a0 tx=0x7f7538006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:08.267 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.265+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f753c014420 con 0x7f75480601e0 2024-02-20T00:30:08.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.470+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75480fbe10 con 0x7f75480601e0 2024-02-20T00:30:08.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.471+0000 7f7536ffd700 1 -- 172.21.15.195:0/3373493036 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f753c010230 con 0x7f75480601e0 2024-02-20T00:30:08.473 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:08.473 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 msgr2=0x7f75300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 0x7f75300323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f75380096a0 tx=0x7f7538006b40).stop 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 msgr2=0x7f75480f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f753c005010 tx=0x7f753c00b040).stop 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 shutdown_connections 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f753002ff10 0x7f75300323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 --2- 172.21.15.195:0/3373493036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f75480601e0 0x7f75480f69a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 >> 172.21.15.195:0/3373493036 conn(0x7f75480f09a0 msgr2=0x7f75480f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 shutdown_connections 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:08.473+0000 7f754759e700 1 -- 172.21.15.195:0/3373493036 wait complete. 2024-02-20T00:30:08.475 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:08.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:06.563963+0000 mgr.smithi087.iydsqo (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:09.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:08.472274+0000 mon.smithi087 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.195:0/3373493036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:10.168 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:10.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:10.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:08.564188+0000 mgr.smithi087.iydsqo (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 -- 172.21.15.195:0/1516212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 msgr2=0x7f43800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:11.893 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 --2- 172.21.15.195:0/1516212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43800f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4374009fb0 tx=0x7f4374009b30).stop 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 -- 172.21.15.195:0/1516212 shutdown_connections 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 --2- 172.21.15.195:0/1516212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43800f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 -- 172.21.15.195:0/1516212 >> 172.21.15.195:0/1516212 conn(0x7f43800f1c70 msgr2=0x7f43800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 -- 172.21.15.195:0/1516212 shutdown_connections 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 -- 172.21.15.195:0/1516212 wait complete. 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.891+0000 7f4385ad0700 1 Processor -- start 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f4385ad0700 1 -- start start 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f4385ad0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:11.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f4385ad0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4380106cd0 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f437f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f437f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57914/0 (socket says 172.21.15.195:57914) 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f437f7fe700 1 -- 172.21.15.195:0/120963623 learned_addr learned my addr 172.21.15.195:0/120963623 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.892+0000 7f437f7fe700 1 -- 172.21.15.195:0/120963623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374005040 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f437f7fe700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4374005010 tx=0x7f437400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4374009320 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4374013380 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4380106ed0 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4374014e50 con 0x7f43800f6510 2024-02-20T00:30:11.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.893+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4380107360 con 0x7f43800f6510 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43740095f0 con 0x7f43800f6510 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4380045ad0 con 0x7f43800f6510 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f4367fff700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 0x7f43680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4374040e90 con 0x7f43800f6510 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f437effd700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 0x7f43680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:11.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.894+0000 7f437effd700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 0x7f43680323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f43700096a0 tx=0x7f4370006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:11.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:11.898+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4374014420 con 0x7f43800f6510 2024-02-20T00:30:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.102+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4380108210 con 0x7f43800f6510 2024-02-20T00:30:12.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.103+0000 7f4367fff700 1 -- 172.21.15.195:0/120963623 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4374010230 con 0x7f43800f6510 2024-02-20T00:30:12.104 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:12.104 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 msgr2=0x7f43680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 0x7f43680323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f43700096a0 tx=0x7f4370006b40).stop 2024-02-20T00:30:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 msgr2=0x7f43801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4374005010 tx=0x7f437400b040).stop 2024-02-20T00:30:12.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 shutdown_connections 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f436802ff10 0x7f43680323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 --2- 172.21.15.195:0/120963623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43800f6510 0x7f43801067e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 >> 172.21.15.195:0/120963623 conn(0x7f43800f1c70 msgr2=0x7f43800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 shutdown_connections 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:12.104+0000 7f4385ad0700 1 -- 172.21.15.195:0/120963623 wait complete. 2024-02-20T00:30:12.107 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:12.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:10.564510+0000 mgr.smithi087.iydsqo (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:12.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:12.103943+0000 mon.smithi087 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.195:0/120963623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:13.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:13.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:14.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:12.564746+0000 mgr.smithi087.iydsqo (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.557+0000 7f85f4521700 1 -- 172.21.15.195:0/1406385928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 msgr2=0x7f85ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.557+0000 7f85f4521700 1 --2- 172.21.15.195:0/1406385928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec0f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f85dc009fb0 tx=0x7f85dc009b30).stop 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- 172.21.15.195:0/1406385928 shutdown_connections 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 --2- 172.21.15.195:0/1406385928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec0f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- 172.21.15.195:0/1406385928 >> 172.21.15.195:0/1406385928 conn(0x7f85ec0f0b70 msgr2=0x7f85ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- 172.21.15.195:0/1406385928 shutdown_connections 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- 172.21.15.195:0/1406385928 wait complete. 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 Processor -- start 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- start start 2024-02-20T00:30:15.561 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f4521700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec060240 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.558+0000 7f85f22bd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f22bd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57934/0 (socket says 172.21.15.195:57934) 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f22bd700 1 -- 172.21.15.195:0/2442050988 learned_addr learned my addr 172.21.15.195:0/2442050988 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f22bd700 1 -- 172.21.15.195:0/2442050988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc005040 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f22bd700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f85dc0089f0 tx=0x7f85dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc009320 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec060440 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85dc013380 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85dc01ed80 con 0x7f85ec0f5410 2024-02-20T00:30:15.562 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.559+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec109b90 con 0x7f85ec0f5410 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.560+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec045ad0 con 0x7f85ec0f5410 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.560+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85dc01e420 con 0x7f85ec0f5410 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.560+0000 7f85e2ffd700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 0x7f85d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.560+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85dc040fe0 con 0x7f85ec0f5410 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.560+0000 7f85f1abc700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 0x7f85d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:15.563 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.561+0000 7f85f1abc700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 0x7f85d8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f85e80096a0 tx=0x7f85e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:15.566 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.565+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85dc0095f0 con 0x7f85ec0f5410 2024-02-20T00:30:15.770 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.768+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85ec10a3a0 con 0x7f85ec0f5410 2024-02-20T00:30:15.771 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.769+0000 7f85e2ffd700 1 -- 172.21.15.195:0/2442050988 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85dc01cb10 con 0x7f85ec0f5410 2024-02-20T00:30:15.771 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:15.771 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:15.773 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 msgr2=0x7f85d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:15.773 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 0x7f85d8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f85e80096a0 tx=0x7f85e8006b40).stop 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 msgr2=0x7f85ec1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f85dc0089f0 tx=0x7f85dc00b040).stop 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 shutdown_connections 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85d802ff60 0x7f85d8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 --2- 172.21.15.195:0/2442050988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85ec0f5410 0x7f85ec1094e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 >> 172.21.15.195:0/2442050988 conn(0x7f85ec0f0b70 msgr2=0x7f85ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:15.774 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 shutdown_connections 2024-02-20T00:30:15.775 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:15.771+0000 7f85f4521700 1 -- 172.21.15.195:0/2442050988 wait complete. 2024-02-20T00:30:15.775 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:16.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:14.565068+0000 mgr.smithi087.iydsqo (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:16.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:15.770356+0000 mon.smithi087 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.195:0/2442050988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:17.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:17.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:18.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:16.565427+0000 mgr.smithi087.iydsqo (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 -- 172.21.15.195:0/198360073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 msgr2=0x7f44280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 --2- 172.21.15.195:0/198360073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44280f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4418009fb0 tx=0x7f4418009b30).stop 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 -- 172.21.15.195:0/198360073 shutdown_connections 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 --2- 172.21.15.195:0/198360073 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44280f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 -- 172.21.15.195:0/198360073 >> 172.21.15.195:0/198360073 conn(0x7f44280f0b70 msgr2=0x7f44280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 -- 172.21.15.195:0/198360073 shutdown_connections 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.376+0000 7f443033e700 1 -- 172.21.15.195:0/198360073 wait complete. 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f443033e700 1 Processor -- start 2024-02-20T00:30:19.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f443033e700 1 -- start start 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f443033e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f443033e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4428106cc0 con 0x7f44280f5410 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f442e0da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f442e0da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:33220/0 (socket says 172.21.15.195:33220) 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f442e0da700 1 -- 172.21.15.195:0/2917538357 learned_addr learned my addr 172.21.15.195:0/2917538357 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f442e0da700 1 -- 172.21.15.195:0/2917538357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418005040 con 0x7f44280f5410 2024-02-20T00:30:19.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.377+0000 7f442e0da700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4418005010 tx=0x7f441800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:19.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4418009320 con 0x7f44280f5410 2024-02-20T00:30:19.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4428106ec0 con 0x7f44280f5410 2024-02-20T00:30:19.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4418013380 con 0x7f44280f5410 2024-02-20T00:30:19.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4418014e50 con 0x7f44280f5410 2024-02-20T00:30:19.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4428107350 con 0x7f44280f5410 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4428045ad0 con 0x7f44280f5410 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.378+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44180095f0 con 0x7f44280f5410 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.379+0000 7f441effd700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 0x7f44140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.379+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4418040e70 con 0x7f44280f5410 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.379+0000 7f442d8d9700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 0x7f44140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:19.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.379+0000 7f442d8d9700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 0x7f44140323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:19.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.383+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4418014420 con 0x7f44280f5410 2024-02-20T00:30:19.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.588+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44281081a0 con 0x7f44280f5410 2024-02-20T00:30:19.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.589+0000 7f441effd700 1 -- 172.21.15.195:0/2917538357 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4418010230 con 0x7f44280f5410 2024-02-20T00:30:19.590 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:19.590 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 msgr2=0x7f44140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 0x7f44140323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).stop 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 msgr2=0x7f44281067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4418005010 tx=0x7f441800b040).stop 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 shutdown_connections 2024-02-20T00:30:19.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f441402ff10 0x7f44140323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:19.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 --2- 172.21.15.195:0/2917538357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44280f5410 0x7f44281067d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:19.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 >> 172.21.15.195:0/2917538357 conn(0x7f44280f0b70 msgr2=0x7f442805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:19.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 shutdown_connections 2024-02-20T00:30:19.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:19.590+0000 7f443033e700 1 -- 172.21.15.195:0/2917538357 wait complete. 2024-02-20T00:30:19.592 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:18.565747+0000 mgr.smithi087.iydsqo (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:20.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:19.588972+0000 mon.smithi087 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.195:0/2917538357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:21.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:21.204 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:20.566020+0000 mgr.smithi087.iydsqo (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 -- 172.21.15.195:0/1356766969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c0f51a0 msgr2=0x7feb1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 --2- 172.21.15.195:0/1356766969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c0f51a0 0x7feb1c0f55c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7feb0c009fb0 tx=0x7feb0c009b30).stop 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 -- 172.21.15.195:0/1356766969 shutdown_connections 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 --2- 172.21.15.195:0/1356766969 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c0f51a0 0x7feb1c0f55c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 -- 172.21.15.195:0/1356766969 >> 172.21.15.195:0/1356766969 conn(0x7feb1c0f09a0 msgr2=0x7feb1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 -- 172.21.15.195:0/1356766969 shutdown_connections 2024-02-20T00:30:22.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.998+0000 7feb1bfff700 1 -- 172.21.15.195:0/1356766969 wait complete. 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1bfff700 1 Processor -- start 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1bfff700 1 -- start start 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1bfff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1c105eb0 con 0x7feb1c1055a0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:33234/0 (socket says 172.21.15.195:33234) 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:22.999+0000 7feb1affd700 1 -- 172.21.15.195:0/3051024147 learned_addr learned my addr 172.21.15.195:0/3051024147 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb1affd700 1 -- 172.21.15.195:0/3051024147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c005040 con 0x7feb1c1055a0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb1affd700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7feb0c009f80 tx=0x7feb0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb0c013380 con 0x7feb1c1055a0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c1060b0 con 0x7feb1c1055a0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb0c013790 con 0x7feb1c1055a0 2024-02-20T00:30:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb0c014420 con 0x7feb1c1055a0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.000+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1c106c00 con 0x7feb1c1055a0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.001+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb0c025060 con 0x7feb1c1055a0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.001+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1c059100 con 0x7feb1c1055a0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.001+0000 7feb137fe700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 0x7feb08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.001+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb0c040ec0 con 0x7feb1c1055a0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.001+0000 7feb13fff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 0x7feb08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:23.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.002+0000 7feb13fff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 0x7feb08032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7feb040096a0 tx=0x7feb04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:23.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.006+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb0c00f280 con 0x7feb1c1055a0 2024-02-20T00:30:23.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.211+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb1c1073c0 con 0x7feb1c1055a0 2024-02-20T00:30:23.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.211+0000 7feb137fe700 1 -- 172.21.15.195:0/3051024147 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb0c00f280 con 0x7feb1c1055a0 2024-02-20T00:30:23.213 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:23.213 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 msgr2=0x7feb08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 0x7feb08032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7feb040096a0 tx=0x7feb04006b40).stop 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 msgr2=0x7feb1c1059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7feb0c009f80 tx=0x7feb0c009b30).stop 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 shutdown_connections 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb0802ff60 0x7feb08032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 --2- 172.21.15.195:0/3051024147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb1c1055a0 0x7feb1c1059c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:23.215 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 >> 172.21.15.195:0/3051024147 conn(0x7feb1c0f09a0 msgr2=0x7feb1c0f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:23.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 shutdown_connections 2024-02-20T00:30:23.216 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:23.213+0000 7feb1bfff700 1 -- 172.21.15.195:0/3051024147 wait complete. 2024-02-20T00:30:23.216 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:24.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:22.566325+0000 mgr.smithi087.iydsqo (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:24.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:23.212796+0000 mon.smithi087 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.195:0/3051024147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:24.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:24.825 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:24.566610+0000 mgr.smithi087.iydsqo (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.529+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1270962521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 msgr2=0x7f1cc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.529+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1270962521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc00f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f1cb0009fb0 tx=0x7f1cb0009b30).stop 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.529+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1270962521 shutdown_connections 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.529+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1270962521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc00f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.530+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1270962521 >> 172.21.15.195:0/1270962521 conn(0x7f1cc00f0b70 msgr2=0x7f1cc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.530+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1270962521 shutdown_connections 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.530+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1270962521 wait complete. 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.530+0000 7f1cc62b7700 1 Processor -- start 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.530+0000 7f1cc62b7700 1 -- start start 2024-02-20T00:30:26.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cc62b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cc62b7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0060240 con 0x7f1cc00f5410 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cbffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cbffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35990/0 (socket says 172.21.15.195:35990) 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cbffff700 1 -- 172.21.15.195:0/1287507019 learned_addr learned my addr 172.21.15.195:0/1287507019 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cbffff700 1 -- 172.21.15.195:0/1287507019 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0005040 con 0x7f1cc00f5410 2024-02-20T00:30:26.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.531+0000 7f1cbffff700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1cb0005010 tx=0x7f1cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb0009320 con 0x7f1cc00f5410 2024-02-20T00:30:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc0060440 con 0x7f1cc00f5410 2024-02-20T00:30:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cb0013380 con 0x7f1cc00f5410 2024-02-20T00:30:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb0014e50 con 0x7f1cc00f5410 2024-02-20T00:30:26.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc0109bd0 con 0x7f1cc00f5410 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cb00095f0 con 0x7f1cc00f5410 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.532+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc0045ad0 con 0x7f1cc00f5410 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.533+0000 7f1cbcff9700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 0x7f1ca80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.533+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cb0040f70 con 0x7f1cc00f5410 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.533+0000 7f1cbf7fe700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 0x7f1ca80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:26.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.533+0000 7f1cbf7fe700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 0x7f1ca80323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f1cb40096a0 tx=0x7f1cb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:26.539 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.537+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cb0014420 con 0x7f1cc00f5410 2024-02-20T00:30:26.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.741+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cc010a3f0 con 0x7f1cc00f5410 2024-02-20T00:30:26.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.742+0000 7f1cbcff9700 1 -- 172.21.15.195:0/1287507019 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1cb001d980 con 0x7f1cc00f5410 2024-02-20T00:30:26.744 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:26.744 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 msgr2=0x7f1ca80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 0x7f1ca80323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f1cb40096a0 tx=0x7f1cb4006b40).stop 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 msgr2=0x7f1cc0109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1cb0005010 tx=0x7f1cb000b040).stop 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 shutdown_connections 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ca802ff10 0x7f1ca80323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 --2- 172.21.15.195:0/1287507019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1cc00f5410 0x7f1cc0109520 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 >> 172.21.15.195:0/1287507019 conn(0x7f1cc00f0b70 msgr2=0x7f1cc00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:26.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 shutdown_connections 2024-02-20T00:30:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:26.743+0000 7f1cc62b7700 1 -- 172.21.15.195:0/1287507019 wait complete. 2024-02-20T00:30:26.746 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:27.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:27 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:26.742848+0000 mon.smithi087 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.195:0/1287507019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:28.384 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:28.384 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:28.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:26.566946+0000 mgr.smithi087.iydsqo (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:30.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- 172.21.15.195:0/595784395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 msgr2=0x7fbae00f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:30.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 --2- 172.21.15.195:0/595784395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae00f5810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fbad4009fb0 tx=0x7fbad4009b30).stop 2024-02-20T00:30:30.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- 172.21.15.195:0/595784395 shutdown_connections 2024-02-20T00:30:30.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 --2- 172.21.15.195:0/595784395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae00f5810 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:30.252 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- 172.21.15.195:0/595784395 >> 172.21.15.195:0/595784395 conn(0x7fbae00f0b70 msgr2=0x7fbae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:30.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- 172.21.15.195:0/595784395 shutdown_connections 2024-02-20T00:30:30.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- 172.21.15.195:0/595784395 wait complete. 2024-02-20T00:30:30.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 Processor -- start 2024-02-20T00:30:30.253 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.250+0000 7fbae70b7700 1 -- start start 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae70b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae70b7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0060240 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae4e53700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae4e53700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36006/0 (socket says 172.21.15.195:36006) 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae4e53700 1 -- 172.21.15.195:0/2500258168 learned_addr learned my addr 172.21.15.195:0/2500258168 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae4e53700 1 -- 172.21.15.195:0/2500258168 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad4005040 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbae4e53700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbad4005010 tx=0x7fbad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.251+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbad4009320 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae0060440 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbad4013380 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbad4014e50 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae0109b70 con 0x7fbae00f53f0 2024-02-20T00:30:30.255 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbad40095f0 con 0x7fbae00f53f0 2024-02-20T00:30:30.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae0045ad0 con 0x7fbae00f53f0 2024-02-20T00:30:30.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.252+0000 7fbadd7fa700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 0x7fbad00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:30.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.253+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbad4040fa0 con 0x7fbae00f53f0 2024-02-20T00:30:30.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.253+0000 7fbadffff700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 0x7fbad00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:30.256 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.253+0000 7fbadffff700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 0x7fbad00323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbacc0096a0 tx=0x7fbacc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:30.258 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.257+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbad4014420 con 0x7fbae00f53f0 2024-02-20T00:30:30.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.459+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbae010a390 con 0x7fbae00f53f0 2024-02-20T00:30:30.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.460+0000 7fbadd7fa700 1 -- 172.21.15.195:0/2500258168 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbad4010230 con 0x7fbae00f53f0 2024-02-20T00:30:30.461 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:30.461 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 msgr2=0x7fbad00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 0x7fbad00323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbacc0096a0 tx=0x7fbacc006b40).stop 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 msgr2=0x7fbae01094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbad4005010 tx=0x7fbad400b040).stop 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 shutdown_connections 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbad002ff10 0x7fbad00323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 --2- 172.21.15.195:0/2500258168 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbae00f53f0 0x7fbae01094c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 >> 172.21.15.195:0/2500258168 conn(0x7fbae00f0b70 msgr2=0x7fbae00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 shutdown_connections 2024-02-20T00:30:30.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:30.461+0000 7fbae70b7700 1 -- 172.21.15.195:0/2500258168 wait complete. 2024-02-20T00:30:30.464 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:30.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:28.567264+0000 mgr.smithi087.iydsqo (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:31.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:30.460923+0000 mon.smithi087 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.195:0/2500258168' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:32.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:32.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:32.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:30.567509+0000 mgr.smithi087.iydsqo (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.848+0000 7ff59846f700 1 -- 172.21.15.195:0/3691940463 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 msgr2=0x7ff5900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.848+0000 7ff59846f700 1 --2- 172.21.15.195:0/3691940463 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff5900f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7ff580009fb0 tx=0x7ff580009b30).stop 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- 172.21.15.195:0/3691940463 shutdown_connections 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 --2- 172.21.15.195:0/3691940463 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff5900f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- 172.21.15.195:0/3691940463 >> 172.21.15.195:0/3691940463 conn(0x7ff5900f1c70 msgr2=0x7ff5900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:33.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- 172.21.15.195:0/3691940463 shutdown_connections 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- 172.21.15.195:0/3691940463 wait complete. 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 Processor -- start 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- start start 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59846f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590069e20 con 0x7ff5900f6510 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59620b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:33.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59620b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:36020/0 (socket says 172.21.15.195:36020) 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.849+0000 7ff59620b700 1 -- 172.21.15.195:0/1652506086 learned_addr learned my addr 172.21.15.195:0/1652506086 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff59620b700 1 -- 172.21.15.195:0/1652506086 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580005040 con 0x7ff5900f6510 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff59620b700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff580005010 tx=0x7ff58000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff580009320 con 0x7ff5900f6510 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590069fc0 con 0x7ff5900f6510 2024-02-20T00:30:33.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff580013380 con 0x7ff5900f6510 2024-02-20T00:30:33.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff580014e50 con 0x7ff5900f6510 2024-02-20T00:30:33.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.850+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59006a450 con 0x7ff5900f6510 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.851+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff590059100 con 0x7ff5900f6510 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.851+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5800095f0 con 0x7ff5900f6510 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.851+0000 7ff586ffd700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 0x7ff57c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.851+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff580040f80 con 0x7ff5900f6510 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.851+0000 7ff595a0a700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 0x7ff57c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:33.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.852+0000 7ff595a0a700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 0x7ff57c0323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff58c0096a0 tx=0x7ff58c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:33.857 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:33.856+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff580014420 con 0x7ff5900f6510 2024-02-20T00:30:34.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.060+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff590045ad0 con 0x7ff5900f6510 2024-02-20T00:30:34.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.060+0000 7ff586ffd700 1 -- 172.21.15.195:0/1652506086 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff580010230 con 0x7ff5900f6510 2024-02-20T00:30:34.062 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:34.062 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 msgr2=0x7ff57c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 0x7ff57c0323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff58c0096a0 tx=0x7ff58c006b40).stop 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 msgr2=0x7ff59006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff580005010 tx=0x7ff58000b040).stop 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 shutdown_connections 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57c02ff10 0x7ff57c0323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 --2- 172.21.15.195:0/1652506086 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5900f6510 0x7ff59006b9a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 >> 172.21.15.195:0/1652506086 conn(0x7ff5900f1c70 msgr2=0x7ff5900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 shutdown_connections 2024-02-20T00:30:34.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:34.062+0000 7ff59846f700 1 -- 172.21.15.195:0/1652506086 wait complete. 2024-02-20T00:30:34.065 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:32.567812+0000 mgr.smithi087.iydsqo (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:34.061673+0000 mon.smithi087 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.195:0/1652506086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:35.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:35.752 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:36.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:34.568124+0000 mgr.smithi087.iydsqo (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 -- 172.21.15.195:0/4286031059 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 msgr2=0x7faa340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 --2- 172.21.15.195:0/4286031059 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa340f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7faa1c009fb0 tx=0x7faa1c009b30).stop 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 -- 172.21.15.195:0/4286031059 shutdown_connections 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 --2- 172.21.15.195:0/4286031059 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa340f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 -- 172.21.15.195:0/4286031059 >> 172.21.15.195:0/4286031059 conn(0x7faa340f1c70 msgr2=0x7faa340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 -- 172.21.15.195:0/4286031059 shutdown_connections 2024-02-20T00:30:37.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 -- 172.21.15.195:0/4286031059 wait complete. 2024-02-20T00:30:37.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.575+0000 7faa39dd6700 1 Processor -- start 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa39dd6700 1 -- start start 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa39dd6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa39dd6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa34106cd0 con 0x7faa340f6510 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa337fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa337fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55102/0 (socket says 172.21.15.195:55102) 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa337fe700 1 -- 172.21.15.195:0/2542468852 learned_addr learned my addr 172.21.15.195:0/2542468852 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.576+0000 7faa337fe700 1 -- 172.21.15.195:0/2542468852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c005040 con 0x7faa340f6510 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa337fe700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7faa1c0089f0 tx=0x7faa1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa1c009320 con 0x7faa340f6510 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34106ed0 con 0x7faa340f6510 2024-02-20T00:30:37.580 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa1c013380 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa1c01ed80 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.577+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa34107360 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa1c01e420 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa34045ad0 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa2bfff700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 0x7faa20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa1c041130 con 0x7faa340f6510 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa32ffd700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 0x7faa20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:37.581 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.578+0000 7faa32ffd700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 0x7faa20032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7faa240096a0 tx=0x7faa24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:37.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.582+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa1c0095f0 con 0x7faa340f6510 2024-02-20T00:30:37.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.787+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa34108210 con 0x7faa340f6510 2024-02-20T00:30:37.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.788+0000 7faa2bfff700 1 -- 172.21.15.195:0/2542468852 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa1c02c2d0 con 0x7faa340f6510 2024-02-20T00:30:37.789 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:37.790 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:37.791 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 msgr2=0x7faa20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 0x7faa20032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7faa240096a0 tx=0x7faa24006b40).stop 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 msgr2=0x7faa341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7faa1c0089f0 tx=0x7faa1c00b040).stop 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 shutdown_connections 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7faa2002ff60 0x7faa20032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:37.792 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 --2- 172.21.15.195:0/2542468852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa340f6510 0x7faa341067e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:37.793 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 >> 172.21.15.195:0/2542468852 conn(0x7faa340f1c70 msgr2=0x7faa340f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:37.793 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 shutdown_connections 2024-02-20T00:30:37.793 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:37.789+0000 7faa39dd6700 1 -- 172.21.15.195:0/2542468852 wait complete. 2024-02-20T00:30:37.793 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:38.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:36.568451+0000 mgr.smithi087.iydsqo (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:38.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:37.788858+0000 mon.smithi087 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.195:0/2542468852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:39.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:39.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:40.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:38.568730+0000 mgr.smithi087.iydsqo (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 -- 172.21.15.195:0/570157770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 msgr2=0x7f25fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 --2- 172.21.15.195:0/570157770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc0f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f25ec009960 tx=0x7f25ec004d10).stop 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 -- 172.21.15.195:0/570157770 shutdown_connections 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 --2- 172.21.15.195:0/570157770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc0f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 -- 172.21.15.195:0/570157770 >> 172.21.15.195:0/570157770 conn(0x7f25fc0f1c70 msgr2=0x7f25fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 -- 172.21.15.195:0/570157770 shutdown_connections 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.226+0000 7f2603443700 1 -- 172.21.15.195:0/570157770 wait complete. 2024-02-20T00:30:41.230 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f2603443700 1 Processor -- start 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f2603443700 1 -- start start 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f2603443700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f2603443700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc106cd0 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f26011df700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f26011df700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55110/0 (socket says 172.21.15.195:55110) 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.227+0000 7f26011df700 1 -- 172.21.15.195:0/3607734866 learned_addr learned my addr 172.21.15.195:0/3607734866 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f26011df700 1 -- 172.21.15.195:0/3607734866 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009640 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f26011df700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f25ec004b70 tx=0x7f25ec004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec018070 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc106ed0 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25ec00b970 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ec01c560 con 0x7f25fc0f6510 2024-02-20T00:30:41.231 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.228+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc107360 con 0x7f25fc0f6510 2024-02-20T00:30:41.232 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.229+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25ec01c6c0 con 0x7f25fc0f6510 2024-02-20T00:30:41.232 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.229+0000 7f25f9ffb700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 0x7f25e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.229+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25ec0412d0 con 0x7f25fc0f6510 2024-02-20T00:30:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.229+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc045ad0 con 0x7f25fc0f6510 2024-02-20T00:30:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.229+0000 7f26009de700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 0x7f25e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:41.233 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.230+0000 7f26009de700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 0x7f25e4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f25f00096a0 tx=0x7f25f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:41.235 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.234+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25ec0127f0 con 0x7f25fc0f6510 2024-02-20T00:30:41.441 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.439+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25fc1080b0 con 0x7f25fc0f6510 2024-02-20T00:30:41.442 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.440+0000 7f25f9ffb700 1 -- 172.21.15.195:0/3607734866 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25ec0127f0 con 0x7f25fc0f6510 2024-02-20T00:30:41.442 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:41.442 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 msgr2=0x7f25e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 0x7f25e4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f25f00096a0 tx=0x7f25f0006b40).stop 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 msgr2=0x7f25fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f25ec004b70 tx=0x7f25ec004d10).stop 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 shutdown_connections 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25e402ff60 0x7f25e4032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 --2- 172.21.15.195:0/3607734866 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25fc0f6510 0x7f25fc1067e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:41.444 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 >> 172.21.15.195:0/3607734866 conn(0x7f25fc0f1c70 msgr2=0x7f25fc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:41.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 shutdown_connections 2024-02-20T00:30:41.445 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:41.442+0000 7f2603443700 1 -- 172.21.15.195:0/3607734866 wait complete. 2024-02-20T00:30:41.445 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:42.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:40.569047+0000 mgr.smithi087.iydsqo (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:42.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:41.441333+0000 mon.smithi087 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.195:0/3607734866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:43.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:43.031 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:42.569346+0000 mgr.smithi087.iydsqo (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 -- 172.21.15.195:0/3996658974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 msgr2=0x7f863c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 --2- 172.21.15.195:0/3996658974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8630009fb0 tx=0x7f8630009b30).stop 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 -- 172.21.15.195:0/3996658974 shutdown_connections 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 --2- 172.21.15.195:0/3996658974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 -- 172.21.15.195:0/3996658974 >> 172.21.15.195:0/3996658974 conn(0x7f863c0f0b70 msgr2=0x7f863c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 -- 172.21.15.195:0/3996658974 shutdown_connections 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.809+0000 7f8643229700 1 -- 172.21.15.195:0/3996658974 wait complete. 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8643229700 1 Processor -- start 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8643229700 1 -- start start 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8643229700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8643229700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c105c00 con 0x7f863c0f5410 2024-02-20T00:30:44.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8640fc5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8640fc5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55136/0 (socket says 172.21.15.195:55136) 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.810+0000 7f8640fc5700 1 -- 172.21.15.195:0/899497349 learned_addr learned my addr 172.21.15.195:0/899497349 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f8640fc5700 1 -- 172.21.15.195:0/899497349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630005040 con 0x7f863c0f5410 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f8640fc5700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8630005010 tx=0x7f863000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8630009320 con 0x7f863c0f5410 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c105e00 con 0x7f863c0f5410 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8630013380 con 0x7f863c0f5410 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8630014e50 con 0x7f863c0f5410 2024-02-20T00:30:44.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.811+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863c106290 con 0x7f863c0f5410 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.812+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86300263a0 con 0x7f863c0f5410 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.812+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863c045ad0 con 0x7f863c0f5410 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.812+0000 7f86397fa700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 0x7f862c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.812+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8630040e40 con 0x7f863c0f5410 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.813+0000 7f863bfff700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 0x7f862c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:44.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.813+0000 7f863bfff700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 0x7f862c0323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f86280096a0 tx=0x7f8628006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:44.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:44.817+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8630012330 con 0x7f863c0f5410 2024-02-20T00:30:45.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.025+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f863c107140 con 0x7f863c0f5410 2024-02-20T00:30:45.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.026+0000 7f86397fa700 1 -- 172.21.15.195:0/899497349 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f863001db70 con 0x7f863c0f5410 2024-02-20T00:30:45.027 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:45.027 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 msgr2=0x7f862c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 0x7f862c0323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f86280096a0 tx=0x7f8628006b40).stop 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 msgr2=0x7f863c105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8630005010 tx=0x7f863000b040).stop 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 shutdown_connections 2024-02-20T00:30:45.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f862c02ff10 0x7f862c0323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 --2- 172.21.15.195:0/899497349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f863c0f5410 0x7f863c105710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 >> 172.21.15.195:0/899497349 conn(0x7f863c0f0b70 msgr2=0x7f863c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 shutdown_connections 2024-02-20T00:30:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:45.027+0000 7f8643229700 1 -- 172.21.15.195:0/899497349 wait complete. 2024-02-20T00:30:45.030 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:45.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:45.026933+0000 mon.smithi087 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.195:0/899497349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:44.569635+0000 mgr.smithi087.iydsqo (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:46.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:46.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.467+0000 7f4936eb4700 1 -- 172.21.15.195:0/2267839127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 msgr2=0x7f49300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.467+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2267839127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49300f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f4924009fb0 tx=0x7f4924009b30).stop 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.467+0000 7f4936eb4700 1 -- 172.21.15.195:0/2267839127 shutdown_connections 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.467+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2267839127 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49300f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.467+0000 7f4936eb4700 1 -- 172.21.15.195:0/2267839127 >> 172.21.15.195:0/2267839127 conn(0x7f49300f0b70 msgr2=0x7f49300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 -- 172.21.15.195:0/2267839127 shutdown_connections 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 -- 172.21.15.195:0/2267839127 wait complete. 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 Processor -- start 2024-02-20T00:30:48.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 -- start start 2024-02-20T00:30:48.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:48.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.468+0000 7f4936eb4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930105bd0 con 0x7f49300f5410 2024-02-20T00:30:48.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4934c50700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4934c50700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44676/0 (socket says 172.21.15.195:44676) 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4934c50700 1 -- 172.21.15.195:0/2097064850 learned_addr learned my addr 172.21.15.195:0/2097064850 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4934c50700 1 -- 172.21.15.195:0/2097064850 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924005040 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4934c50700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4924005010 tx=0x7f492400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4924009320 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4930105dd0 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4924013380 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4924014e50 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.469+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4930106260 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.470+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4924014420 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.470+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4930045ad0 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.470+0000 7f492d7fa700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 0x7f49200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.470+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4924040f00 con 0x7f49300f5410 2024-02-20T00:30:48.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.470+0000 7f492ffff700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 0x7f49200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:48.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.471+0000 7f492ffff700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 0x7f49200323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f491c0096a0 tx=0x7f491c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:48.476 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.475+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4924014790 con 0x7f49300f5410 2024-02-20T00:30:48.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:46.569877+0000 mgr.smithi087.iydsqo (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:48.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.679+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4930107110 con 0x7f49300f5410 2024-02-20T00:30:48.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.680+0000 7f492d7fa700 1 -- 172.21.15.195:0/2097064850 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f492401d380 con 0x7f49300f5410 2024-02-20T00:30:48.682 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:48.682 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:48.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.681+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 msgr2=0x7f49200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 0x7f49200323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f491c0096a0 tx=0x7f491c006b40).stop 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 msgr2=0x7f49301056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4924005010 tx=0x7f492400b040).stop 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 shutdown_connections 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f492002ff10 0x7f49200323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 --2- 172.21.15.195:0/2097064850 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49300f5410 0x7f49301056e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 >> 172.21.15.195:0/2097064850 conn(0x7f49300f0b70 msgr2=0x7f49300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 shutdown_connections 2024-02-20T00:30:48.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:48.682+0000 7f4936eb4700 1 -- 172.21.15.195:0/2097064850 wait complete. 2024-02-20T00:30:48.685 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:49.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:48.681326+0000 mon.smithi087 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.195:0/2097064850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:50.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:50.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:50.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:48.570208+0000 mgr.smithi087.iydsqo (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/1087179027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 msgr2=0x7f6c580f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/1087179027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c580f68a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6c4c009fb0 tx=0x7f6c4c009b30).stop 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/1087179027 shutdown_connections 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/1087179027 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c580f68a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/1087179027 >> 172.21.15.195:0/1087179027 conn(0x7f6c580f1c00 msgr2=0x7f6c580f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/1087179027 shutdown_connections 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.139+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/1087179027 wait complete. 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c5dc5b700 1 Processor -- start 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c5dc5b700 1 -- start start 2024-02-20T00:30:52.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c5dc5b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c5dc5b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c58106ce0 con 0x7f6c580f6480 2024-02-20T00:30:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c577fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44690/0 (socket says 172.21.15.195:44690) 2024-02-20T00:30:52.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.140+0000 7f6c577fe700 1 -- 172.21.15.195:0/337473920 learned_addr learned my addr 172.21.15.195:0/337473920 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:52.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c577fe700 1 -- 172.21.15.195:0/337473920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c005040 con 0x7f6c580f6480 2024-02-20T00:30:52.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c577fe700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6c4c005010 tx=0x7f6c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:52.144 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c4c009320 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c4c013380 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c58106ee0 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c4c014e50 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.141+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c58107370 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.142+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c4c014420 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.142+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c58045ad0 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.142+0000 7f6c3ffff700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 0x7f6c400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.142+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c4c040f30 con 0x7f6c580f6480 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.142+0000 7f6c56ffd700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 0x7f6c400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:52.145 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.143+0000 7f6c56ffd700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 0x7f6c400323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6c480096a0 tx=0x7f6c48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:52.148 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.147+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c4c0095f0 con 0x7f6c580f6480 2024-02-20T00:30:52.355 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.354+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c580fc090 con 0x7f6c580f6480 2024-02-20T00:30:52.356 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.354+0000 7f6c3ffff700 1 -- 172.21.15.195:0/337473920 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c4c01db70 con 0x7f6c580f6480 2024-02-20T00:30:52.356 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:52.356 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 msgr2=0x7f6c400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 0x7f6c400323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6c480096a0 tx=0x7f6c48006b40).stop 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 msgr2=0x7f6c581067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6c4c005010 tx=0x7f6c4c00b040).stop 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 shutdown_connections 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f6c4002ff10 0x7f6c400323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:52.358 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 --2- 172.21.15.195:0/337473920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c580f6480 0x7f6c581067f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 >> 172.21.15.195:0/337473920 conn(0x7f6c580f1c00 msgr2=0x7f6c580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 shutdown_connections 2024-02-20T00:30:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:52.356+0000 7f6c5dc5b700 1 -- 172.21.15.195:0/337473920 wait complete. 2024-02-20T00:30:52.359 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:52.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:50.570493+0000 mgr.smithi087.iydsqo (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:53.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:52.355829+0000 mon.smithi087 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.195:0/337473920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:53.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:53.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:52.570731+0000 mgr.smithi087.iydsqo (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- 172.21.15.195:0/507530149 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 msgr2=0x7fa6b00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/507530149 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f8f80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fa6ac009fb0 tx=0x7fa6ac009b30).stop 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- 172.21.15.195:0/507530149 shutdown_connections 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/507530149 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f8f80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- 172.21.15.195:0/507530149 >> 172.21.15.195:0/507530149 conn(0x7fa6b00f09a0 msgr2=0x7fa6b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- 172.21.15.195:0/507530149 shutdown_connections 2024-02-20T00:30:55.631 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- 172.21.15.195:0/507530149 wait complete. 2024-02-20T00:30:55.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 Processor -- start 2024-02-20T00:30:55.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- start start 2024-02-20T00:30:55.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:55.632 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.629+0000 7fa6b65ca700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00f4e80 con 0x7fa6b0060180 2024-02-20T00:30:55.633 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b55c8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b55c8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44712/0 (socket says 172.21.15.195:44712) 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b55c8700 1 -- 172.21.15.195:0/483626834 learned_addr learned my addr 172.21.15.195:0/483626834 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b55c8700 1 -- 172.21.15.195:0/483626834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac005040 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b55c8700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fa6ac005010 tx=0x7fa6ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ac009320 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00f5020 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6ac013380 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.630+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ac014e50 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.631+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b00f54b0 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.631+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6ac014420 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.631+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b0059100 con 0x7fa6b0060180 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.631+0000 7fa6a5ffb700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 0x7fa69c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:55.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.632+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6ac040e60 con 0x7fa6b0060180 2024-02-20T00:30:55.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.632+0000 7fa6b4dc7700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 0x7fa69c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:55.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.632+0000 7fa6b4dc7700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 0x7fa69c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:55.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.636+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6ac0095f0 con 0x7fa6b0060180 2024-02-20T00:30:55.843 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.841+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6b00fbe20 con 0x7fa6b0060180 2024-02-20T00:30:55.844 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.842+0000 7fa6a5ffb700 1 -- 172.21.15.195:0/483626834 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6ac01db70 con 0x7fa6b0060180 2024-02-20T00:30:55.844 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:55.844 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 msgr2=0x7fa69c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 0x7fa69c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).stop 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 msgr2=0x7fa6b00f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fa6ac005010 tx=0x7fa6ac00b040).stop 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 shutdown_connections 2024-02-20T00:30:55.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa69c02ff10 0x7fa69c0323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:55.847 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 --2- 172.21.15.195:0/483626834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa6b0060180 0x7fa6b00f6a00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:55.847 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 >> 172.21.15.195:0/483626834 conn(0x7fa6b00f09a0 msgr2=0x7fa6b00f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:55.847 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 shutdown_connections 2024-02-20T00:30:55.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:55.843+0000 7fa6b65ca700 1 -- 172.21.15.195:0/483626834 wait complete. 2024-02-20T00:30:55.848 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:30:56.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:54.571004+0000 mgr.smithi087.iydsqo (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:55.843003+0000 mon.smithi087 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.195:0/483626834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:30:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:56.668512+0000 mon.smithi087 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:30:57.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:56.669548+0000 mon.smithi087 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:30:57.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:30:57.541 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:30:58.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:30:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:56.571345+0000 mgr.smithi087.iydsqo (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/4276588544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 msgr2=0x7ff5a00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/4276588544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a00f5750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7ff590009fb0 tx=0x7ff590009b30).stop 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/4276588544 shutdown_connections 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/4276588544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a00f5750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/4276588544 >> 172.21.15.195:0/4276588544 conn(0x7ff5a00f0a90 msgr2=0x7ff5a00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/4276588544 shutdown_connections 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.296+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/4276588544 wait complete. 2024-02-20T00:30:59.318 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a7ea0700 1 Processor -- start 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a7ea0700 1 -- start start 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a7ea0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a7ea0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a0105e70 con 0x7ff5a00f5330 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a5c3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a5c3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46514/0 (socket says 172.21.15.195:46514) 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.297+0000 7ff5a5c3c700 1 -- 172.21.15.195:0/897416381 learned_addr learned my addr 172.21.15.195:0/897416381 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5a5c3c700 1 -- 172.21.15.195:0/897416381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590005040 con 0x7ff5a00f5330 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5a5c3c700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff590005010 tx=0x7ff59000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff590009320 con 0x7ff5a00f5330 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff590013380 con 0x7ff5a00f5330 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a0106070 con 0x7ff5a00f5330 2024-02-20T00:30:59.319 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff590014e50 con 0x7ff5a00f5330 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.298+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a0106500 con 0x7ff5a00f5330 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5900095f0 con 0x7ff5a00f5330 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a0045ad0 con 0x7ff5a00f5330 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5967fc700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 0x7ff58c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff590040f60 con 0x7ff5a00f5330 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5a543b700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 0x7ff58c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.299+0000 7ff5a543b700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 0x7ff58c0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff59c0096a0 tx=0x7ff59c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:30:59.320 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.303+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff590014420 con 0x7ff5a00f5330 2024-02-20T00:30:59.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.508+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5a005ea50 con 0x7ff5a00f5330 2024-02-20T00:30:59.510 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.509+0000 7ff5967fc700 1 -- 172.21.15.195:0/897416381 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff590010230 con 0x7ff5a00f5330 2024-02-20T00:30:59.510 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:30:59.511 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 msgr2=0x7ff58c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 0x7ff58c0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff59c0096a0 tx=0x7ff59c006b40).stop 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 msgr2=0x7ff5a0105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff590005010 tx=0x7ff59000b040).stop 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 shutdown_connections 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff58c02ff10 0x7ff58c0323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:59.512 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 --2- 172.21.15.195:0/897416381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5a00f5330 0x7ff5a0105980 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:30:59.513 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 >> 172.21.15.195:0/897416381 conn(0x7ff5a00f0a90 msgr2=0x7ff5a00f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:30:59.513 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 shutdown_connections 2024-02-20T00:30:59.513 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:30:59.510+0000 7ff5a7ea0700 1 -- 172.21.15.195:0/897416381 wait complete. 2024-02-20T00:30:59.513 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:00.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:30:58.571644+0000 mgr.smithi087.iydsqo (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:00.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:30:59.510002+0000 mon.smithi087 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.195:0/897416381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:01.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:01.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:02.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:00.571927+0000 mgr.smithi087.iydsqo (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 -- 172.21.15.195:0/1859184561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 msgr2=0x7f05bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 --2- 172.21.15.195:0/1859184561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f05ac009fb0 tx=0x7f05ac009b30).stop 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 -- 172.21.15.195:0/1859184561 shutdown_connections 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 --2- 172.21.15.195:0/1859184561 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 -- 172.21.15.195:0/1859184561 >> 172.21.15.195:0/1859184561 conn(0x7f05bc0f0b70 msgr2=0x7f05bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:02.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 -- 172.21.15.195:0/1859184561 shutdown_connections 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.944+0000 7f05c1766700 1 -- 172.21.15.195:0/1859184561 wait complete. 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.945+0000 7f05c1766700 1 Processor -- start 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.945+0000 7f05c1766700 1 -- start start 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.945+0000 7f05c1766700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.945+0000 7f05c1766700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0fb960 con 0x7f05bc0f5410 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05baffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05baffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46536/0 (socket says 172.21.15.195:46536) 2024-02-20T00:31:02.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05baffd700 1 -- 172.21.15.195:0/2029799723 learned_addr learned my addr 172.21.15.195:0/2029799723 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:02.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05baffd700 1 -- 172.21.15.195:0/2029799723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac005040 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05baffd700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f05ac005010 tx=0x7f05ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac009320 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc0fbb00 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05ac013380 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05ac014e50 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.946+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc0fbf90 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.947+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05ac014420 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.947+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc059100 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.947+0000 7f05a37fe700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.947+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05ac040f40 con 0x7f05bc0f5410 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.948+0000 7f05ba7fc700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:02.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.948+0000 7f05ba7fc700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 0x7f05a40323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f05b00096a0 tx=0x7f05b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:02.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:02.952+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05ac0095f0 con 0x7f05bc0f5410 2024-02-20T00:31:03.158 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.156+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05bc0fafa0 con 0x7f05bc0f5410 2024-02-20T00:31:03.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.157+0000 7f05a37fe700 1 -- 172.21.15.195:0/2029799723 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05ac01db70 con 0x7f05bc0f5410 2024-02-20T00:31:03.159 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:03.159 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:03.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 msgr2=0x7f05a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 0x7f05a40323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f05b00096a0 tx=0x7f05b0006b40).stop 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 msgr2=0x7f05bc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f05ac005010 tx=0x7f05ac00b040).stop 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 shutdown_connections 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 --2- 172.21.15.195:0/2029799723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05bc0f5410 0x7f05bc0fd4e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 >> 172.21.15.195:0/2029799723 conn(0x7f05bc0f0b70 msgr2=0x7f05bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 shutdown_connections 2024-02-20T00:31:03.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:03.159+0000 7f05c1766700 1 -- 172.21.15.195:0/2029799723 wait complete. 2024-02-20T00:31:03.162 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:04.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:02.572218+0000 mgr.smithi087.iydsqo (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:04.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:03.158227+0000 mon.smithi087 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.195:0/2029799723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:04.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:04.862 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:06.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:04.572493+0000 mgr.smithi087.iydsqo (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- 172.21.15.195:0/3148363249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 msgr2=0x7f36c40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 --2- 172.21.15.195:0/3148363249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c40f66c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f36b4009fb0 tx=0x7f36b4009b30).stop 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- 172.21.15.195:0/3148363249 shutdown_connections 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 --2- 172.21.15.195:0/3148363249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c40f66c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- 172.21.15.195:0/3148363249 >> 172.21.15.195:0/3148363249 conn(0x7f36c40f1aa0 msgr2=0x7f36c40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- 172.21.15.195:0/3148363249 shutdown_connections 2024-02-20T00:31:06.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- 172.21.15.195:0/3148363249 wait complete. 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 Processor -- start 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.631+0000 7f36c898e700 1 -- start start 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c898e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c898e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4069e20 con 0x7f36c40f62a0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44808/0 (socket says 172.21.15.195:44808) 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c37fe700 1 -- 172.21.15.195:0/505747093 learned_addr learned my addr 172.21.15.195:0/505747093 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c37fe700 1 -- 172.21.15.195:0/505747093 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4005040 con 0x7f36c40f62a0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c37fe700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f36b40089f0 tx=0x7f36b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36b4009320 con 0x7f36c40f62a0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4069fc0 con 0x7f36c40f62a0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36b4013380 con 0x7f36c40f62a0 2024-02-20T00:31:06.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36b4014e20 con 0x7f36c40f62a0 2024-02-20T00:31:06.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.632+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c406a450 con 0x7f36c40f62a0 2024-02-20T00:31:06.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.633+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36b4014420 con 0x7f36c40f62a0 2024-02-20T00:31:06.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.633+0000 7f36abfff700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 0x7f36ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:06.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.633+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4059100 con 0x7f36c40f62a0 2024-02-20T00:31:06.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.633+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36b4040fc0 con 0x7f36c40f62a0 2024-02-20T00:31:06.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.633+0000 7f36c2ffd700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 0x7f36ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:06.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.634+0000 7f36c2ffd700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 0x7f36ac0323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f36b80096a0 tx=0x7f36b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:06.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.638+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36b40095f0 con 0x7f36c40f62a0 2024-02-20T00:31:06.846 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.844+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36c40fbe30 con 0x7f36c40f62a0 2024-02-20T00:31:06.847 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.846+0000 7f36abfff700 1 -- 172.21.15.195:0/505747093 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36b4007dd0 con 0x7f36c40f62a0 2024-02-20T00:31:06.847 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:06.847 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:06.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 msgr2=0x7f36ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:06.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 0x7f36ac0323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f36b80096a0 tx=0x7f36b8006b40).stop 2024-02-20T00:31:06.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 msgr2=0x7f36c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:06.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f36b40089f0 tx=0x7f36b400b040).stop 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 shutdown_connections 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f36ac02ff10 0x7f36ac0323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 --2- 172.21.15.195:0/505747093 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36c40f62a0 0x7f36c406b9a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 >> 172.21.15.195:0/505747093 conn(0x7f36c40f1aa0 msgr2=0x7f36c40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 shutdown_connections 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:06.847+0000 7f36c898e700 1 -- 172.21.15.195:0/505747093 wait complete. 2024-02-20T00:31:06.850 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:07.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:06.567204+0000 mon.smithi087 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi087", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T00:31:07.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:06.570026+0000 mon.smithi087 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi087", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T00:31:07.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:06.570806+0000 mon.smithi087 (mon.0) 203 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:31:07.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:06.846569+0000 mon.smithi087 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.195:0/505747093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:31:06.565847+0000 mgr.smithi087.iydsqo (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi087 ... 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:26:45Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:27:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:28:00Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:31:08.283 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:29:14Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:29:51Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:30:29Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:31:08.284 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('uid/gid not found') 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: uid/gid not found 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: yield (conn, connr) 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: code, '\n'.join(err))) 2024-02-20T00:31:08.285 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi087 ... 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:26:45Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:27:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:28:00Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:29:14Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:29:51Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:30:29Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:31:08.286 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('uid/gid not found') 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: uid/gid not found 2024-02-20T00:31:08.287 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:31:06.571510+0000 mgr.smithi087.iydsqo (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi087 on smithi087 2024-02-20T00:31:08.288 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:06.572704+0000 mgr.smithi087.iydsqo (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:08.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:08.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:10.175 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- 172.21.15.195:0/1963654904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 msgr2=0x7f20040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:10.175 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 --2- 172.21.15.195:0/1963654904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20040f55c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f1ff4009fb0 tx=0x7f1ff4009b30).stop 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- 172.21.15.195:0/1963654904 shutdown_connections 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 --2- 172.21.15.195:0/1963654904 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20040f55c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- 172.21.15.195:0/1963654904 >> 172.21.15.195:0/1963654904 conn(0x7f20040f09a0 msgr2=0x7f20040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- 172.21.15.195:0/1963654904 shutdown_connections 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- 172.21.15.195:0/1963654904 wait complete. 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 Processor -- start 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- start start 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.173+0000 7f200359e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2004060220 con 0x7f20040f51a0 2024-02-20T00:31:10.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44814/0 (socket says 172.21.15.195:44814) 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200259c700 1 -- 172.21.15.195:0/1944689039 learned_addr learned my addr 172.21.15.195:0/1944689039 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200259c700 1 -- 172.21.15.195:0/1944689039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff4005040 con 0x7f20040f51a0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200259c700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1ff40089f0 tx=0x7f1ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff4009320 con 0x7f20040f51a0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004060420 con 0x7f20040f51a0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ff4013380 con 0x7f20040f51a0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.174+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff4014e20 con 0x7f20040f51a0 2024-02-20T00:31:10.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.175+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20041099a0 con 0x7f20040f51a0 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.175+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ff40095f0 con 0x7f20040f51a0 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.175+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2004059100 con 0x7f20040f51a0 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.175+0000 7f1ff2ffd700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.175+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ff4040f50 con 0x7f20040f51a0 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.176+0000 7f2001d9b700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:10.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.176+0000 7f2001d9b700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 0x7f1fec032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f1ff80096a0 tx=0x7f1ff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:10.181 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.180+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ff4014420 con 0x7f20040f51a0 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:08.572944+0000 mgr.smithi087.iydsqo (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.826936+0000 mon.smithi087 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.894026+0000 mon.smithi087 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.898967+0000 mon.smithi087 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.899351+0000 mon.smithi087 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.899776+0000 mgr.smithi087.iydsqo (mgr.14162) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:08.905309+0000 mon.smithi087 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:31:10.282 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:09 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:31:08.908105+0000 mgr.smithi087.iydsqo (mgr.14162) 161 : cephadm [INF] Deploying daemon grafana.smithi087 on smithi087 2024-02-20T00:31:10.386 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.385+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2004045ad0 con 0x7f20040f51a0 2024-02-20T00:31:10.387 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.386+0000 7f1ff2ffd700 1 -- 172.21.15.195:0/1944689039 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ff401d980 con 0x7f20040f51a0 2024-02-20T00:31:10.387 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:10.387 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:10.388 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 msgr2=0x7f1fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:10.388 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 0x7f1fec032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f1ff80096a0 tx=0x7f1ff8006b40).stop 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 msgr2=0x7f20041092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1ff40089f0 tx=0x7f1ff400b040).stop 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 shutdown_connections 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1fec02ff60 0x7f1fec032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 --2- 172.21.15.195:0/1944689039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f20040f51a0 0x7f20041092f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 >> 172.21.15.195:0/1944689039 conn(0x7f20040f09a0 msgr2=0x7f20040f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 shutdown_connections 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:10.388+0000 7f200359e700 1 -- 172.21.15.195:0/1944689039 wait complete. 2024-02-20T00:31:10.389 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:11.281 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:10.386256+0000 mon.smithi087 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.195:0/1944689039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:12.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:12.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:12.281 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:10.573153+0000 mgr.smithi087.iydsqo (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:13.725 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- 172.21.15.195:0/3121290075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 msgr2=0x7eff680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 --2- 172.21.15.195:0/3121290075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff680f55c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7eff58009fb0 tx=0x7eff58009b30).stop 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- 172.21.15.195:0/3121290075 shutdown_connections 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 --2- 172.21.15.195:0/3121290075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff680f55c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- 172.21.15.195:0/3121290075 >> 172.21.15.195:0/3121290075 conn(0x7eff680f09a0 msgr2=0x7eff680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- 172.21.15.195:0/3121290075 shutdown_connections 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- 172.21.15.195:0/3121290075 wait complete. 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 Processor -- start 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.718+0000 7eff6759e700 1 -- start start 2024-02-20T00:31:13.726 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6759e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff68060280 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:44832/0 (socket says 172.21.15.195:44832) 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6659c700 1 -- 172.21.15.195:0/442510309 learned_addr learned my addr 172.21.15.195:0/442510309 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6659c700 1 -- 172.21.15.195:0/442510309 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58005040 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6659c700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7eff580089f0 tx=0x7eff5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff58009320 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff68060480 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff58013380 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.719+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff58014e20 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.720+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff681099c0 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.720+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff580263a0 con 0x7eff680f51a0 2024-02-20T00:31:13.727 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.720+0000 7eff5effd700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 0x7eff540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:13.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.720+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff58040ee0 con 0x7eff680f51a0 2024-02-20T00:31:13.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.720+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff68045ad0 con 0x7eff680f51a0 2024-02-20T00:31:13.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.721+0000 7eff65d9b700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 0x7eff540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:13.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.721+0000 7eff65d9b700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 0x7eff540323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7eff500096a0 tx=0x7eff50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:13.728 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.725+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff58012920 con 0x7eff680f51a0 2024-02-20T00:31:13.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.928+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff6810a070 con 0x7eff680f51a0 2024-02-20T00:31:13.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.929+0000 7eff5effd700 1 -- 172.21.15.195:0/442510309 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff5802b020 con 0x7eff680f51a0 2024-02-20T00:31:13.930 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:13.930 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 msgr2=0x7eff540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 0x7eff540323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7eff500096a0 tx=0x7eff50006b40).stop 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 msgr2=0x7eff68109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7eff580089f0 tx=0x7eff5800b040).stop 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 shutdown_connections 2024-02-20T00:31:13.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff5402ff10 0x7eff540323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 --2- 172.21.15.195:0/442510309 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff680f51a0 0x7eff68109290 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 >> 172.21.15.195:0/442510309 conn(0x7eff680f09a0 msgr2=0x7eff680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 shutdown_connections 2024-02-20T00:31:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:13.930+0000 7eff6759e700 1 -- 172.21.15.195:0/442510309 wait complete. 2024-02-20T00:31:13.932 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:12.573496+0000 mgr.smithi087.iydsqo (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:14.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:13.929091+0000 mon.smithi087 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.195:0/442510309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:15.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:15.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:16.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:14.573854+0000 mgr.smithi087.iydsqo (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:17.291 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- 172.21.15.195:0/1817695928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 msgr2=0x7fad780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:17.291 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/1817695928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad780f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fad6c009fb0 tx=0x7fad6c009b30).stop 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- 172.21.15.195:0/1817695928 shutdown_connections 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/1817695928 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad780f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- 172.21.15.195:0/1817695928 >> 172.21.15.195:0/1817695928 conn(0x7fad780f0b70 msgr2=0x7fad780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- 172.21.15.195:0/1817695928 shutdown_connections 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- 172.21.15.195:0/1817695928 wait complete. 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 Processor -- start 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.289+0000 7fad7ebb1700 1 -- start start 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7ebb1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7ebb1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78105bd0 con 0x7fad780f5410 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7c94d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7c94d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34432/0 (socket says 172.21.15.195:34432) 2024-02-20T00:31:17.292 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7c94d700 1 -- 172.21.15.195:0/2136660524 learned_addr learned my addr 172.21.15.195:0/2136660524 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7c94d700 1 -- 172.21.15.195:0/2136660524 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c005040 con 0x7fad780f5410 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad7c94d700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fad6c005010 tx=0x7fad6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.290+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad6c009320 con 0x7fad780f5410 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad78105dd0 con 0x7fad780f5410 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad6c013380 con 0x7fad780f5410 2024-02-20T00:31:17.294 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad6c014e50 con 0x7fad780f5410 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78106260 con 0x7fad780f5410 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad6c0263a0 con 0x7fad780f5410 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad757fa700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad6c040f00 con 0x7fad780f5410 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.291+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad78059100 con 0x7fad780f5410 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.292+0000 7fad77fff700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:17.295 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.292+0000 7fad77fff700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 0x7fad680323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fad640096a0 tx=0x7fad64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:17.297 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.296+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad6c026630 con 0x7fad780f5410 2024-02-20T00:31:17.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.501+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad780fafa0 con 0x7fad780f5410 2024-02-20T00:31:17.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.501+0000 7fad757fa700 1 -- 172.21.15.195:0/2136660524 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad6c02b020 con 0x7fad780f5410 2024-02-20T00:31:17.503 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:17.503 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 msgr2=0x7fad680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 0x7fad680323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fad640096a0 tx=0x7fad64006b40).stop 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 msgr2=0x7fad781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fad6c005010 tx=0x7fad6c00b040).stop 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 shutdown_connections 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 --2- 172.21.15.195:0/2136660524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad780f5410 0x7fad781056e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 >> 172.21.15.195:0/2136660524 conn(0x7fad780f0b70 msgr2=0x7fad780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 shutdown_connections 2024-02-20T00:31:17.505 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:17.503+0000 7fad7ebb1700 1 -- 172.21.15.195:0/2136660524 wait complete. 2024-02-20T00:31:17.506 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:16.574043+0000 mgr.smithi087.iydsqo (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:17.502522+0000 mon.smithi087 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.195:0/2136660524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:19.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:19.358 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:20.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:18.574278+0000 mgr.smithi087.iydsqo (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 -- 172.21.15.195:0/4262042025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 msgr2=0x7f88940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 --2- 172.21.15.195:0/4262042025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88940f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f8888009fb0 tx=0x7f8888009b30).stop 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 -- 172.21.15.195:0/4262042025 shutdown_connections 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 --2- 172.21.15.195:0/4262042025 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88940f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 -- 172.21.15.195:0/4262042025 >> 172.21.15.195:0/4262042025 conn(0x7f88940f0b70 msgr2=0x7f88940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 -- 172.21.15.195:0/4262042025 shutdown_connections 2024-02-20T00:31:21.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.202+0000 7f8899922700 1 -- 172.21.15.195:0/4262042025 wait complete. 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8899922700 1 Processor -- start 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8899922700 1 -- start start 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8899922700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8899922700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8894060240 con 0x7f88940f5410 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34444/0 (socket says 172.21.15.195:34444) 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8892ffd700 1 -- 172.21.15.195:0/948399635 learned_addr learned my addr 172.21.15.195:0/948399635 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8892ffd700 1 -- 172.21.15.195:0/948399635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888005040 con 0x7f88940f5410 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.203+0000 7f8892ffd700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f8888005010 tx=0x7f888800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.204+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8888009320 con 0x7f88940f5410 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.204+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8888013380 con 0x7f88940f5410 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.204+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8894060440 con 0x7f88940f5410 2024-02-20T00:31:21.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.204+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8888014e50 con 0x7f88940f5410 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.204+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8894109b90 con 0x7f88940f5410 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88880095f0 con 0x7f88940f5410 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8894045ad0 con 0x7f88940f5410 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f887b7fe700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 0x7f887c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8888040f60 con 0x7f88940f5410 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f88927fc700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 0x7f887c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:21.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.205+0000 7f88927fc700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 0x7f887c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f88840096a0 tx=0x7f8884006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:21.211 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.210+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8888014420 con 0x7f88940f5410 2024-02-20T00:31:21.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.414+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f889410a3b0 con 0x7f88940f5410 2024-02-20T00:31:21.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.414+0000 7f887b7fe700 1 -- 172.21.15.195:0/948399635 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f888801d980 con 0x7f88940f5410 2024-02-20T00:31:21.416 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:21.416 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:21.417 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 msgr2=0x7f887c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 0x7f887c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f88840096a0 tx=0x7f8884006b40).stop 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 msgr2=0x7f88941094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f8888005010 tx=0x7f888800b040).stop 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 shutdown_connections 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f887c02ff10 0x7f887c0323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 --2- 172.21.15.195:0/948399635 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88940f5410 0x7f88941094e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 >> 172.21.15.195:0/948399635 conn(0x7f88940f0b70 msgr2=0x7f88940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 shutdown_connections 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:21.416+0000 7f8899922700 1 -- 172.21.15.195:0/948399635 wait complete. 2024-02-20T00:31:21.418 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:22.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:20.574491+0000 mgr.smithi087.iydsqo (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:21.415662+0000 mon.smithi087 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.195:0/948399635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:23.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:23.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:24.357 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:22.574705+0000 mgr.smithi087.iydsqo (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:24.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 -- 172.21.15.195:0/2352877476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 msgr2=0x7f1f640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:24.984 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 --2- 172.21.15.195:0/2352877476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f640f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f1f54009fb0 tx=0x7f1f54009b30).stop 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 -- 172.21.15.195:0/2352877476 shutdown_connections 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 --2- 172.21.15.195:0/2352877476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f640f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 -- 172.21.15.195:0/2352877476 >> 172.21.15.195:0/2352877476 conn(0x7f1f640f09a0 msgr2=0x7f1f640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 -- 172.21.15.195:0/2352877476 shutdown_connections 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.982+0000 7f1f63fff700 1 -- 172.21.15.195:0/2352877476 wait complete. 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f63fff700 1 Processor -- start 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f63fff700 1 -- start start 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f63fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:24.985 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f63fff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f64060280 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f62ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f62ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34456/0 (socket says 172.21.15.195:34456) 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.983+0000 7f1f62ffd700 1 -- 172.21.15.195:0/3567472992 learned_addr learned my addr 172.21.15.195:0/3567472992 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f62ffd700 1 -- 172.21.15.195:0/3567472992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54005040 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f62ffd700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f1f540089f0 tx=0x7f1f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f54009320 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f64060480 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f54013380 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f54014e20 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.984+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f641099c0 con 0x7f1f640f51a0 2024-02-20T00:31:24.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.985+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f54014420 con 0x7f1f640f51a0 2024-02-20T00:31:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.985+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f64045ad0 con 0x7f1f640f51a0 2024-02-20T00:31:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.985+0000 7f1f4b7fe700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 0x7f1f4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.985+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f54040f80 con 0x7f1f640f51a0 2024-02-20T00:31:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.985+0000 7f1f627fc700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 0x7f1f4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:24.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.986+0000 7f1f627fc700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 0x7f1f4c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f580096a0 tx=0x7f1f58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:24.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:24.990+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f5406d2b0 con 0x7f1f640f51a0 2024-02-20T00:31:25.197 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.196+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f640fad30 con 0x7f1f640f51a0 2024-02-20T00:31:25.198 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.196+0000 7f1f4b7fe700 1 -- 172.21.15.195:0/3567472992 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f540095f0 con 0x7f1f640f51a0 2024-02-20T00:31:25.199 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:25.199 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:25.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 msgr2=0x7f1f4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 0x7f1f4c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f580096a0 tx=0x7f1f58006b40).stop 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 msgr2=0x7f1f64109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f1f540089f0 tx=0x7f1f5400b040).stop 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 shutdown_connections 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1f4c02ff10 0x7f1f4c0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 --2- 172.21.15.195:0/3567472992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f640f51a0 0x7f1f64109290 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 >> 172.21.15.195:0/3567472992 conn(0x7f1f640f09a0 msgr2=0x7f1f640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:25.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 shutdown_connections 2024-02-20T00:31:25.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:25.198+0000 7f1f63fff700 1 -- 172.21.15.195:0/3567472992 wait complete. 2024-02-20T00:31:25.204 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:26.486 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:24.575040+0000 mgr.smithi087.iydsqo (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:26.487 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:25.197685+0000 mon.smithi087 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.195:0/3567472992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:26.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:26.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.710+0000 7fed5e48a700 1 -- 172.21.15.195:0/1317570164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 msgr2=0x7fed580f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.710+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1317570164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed580f4b60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fed48009fb0 tx=0x7fed48009b30).stop 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- 172.21.15.195:0/1317570164 shutdown_connections 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1317570164 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed580f4b60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- 172.21.15.195:0/1317570164 >> 172.21.15.195:0/1317570164 conn(0x7fed580f0260 msgr2=0x7fed580f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- 172.21.15.195:0/1317570164 shutdown_connections 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- 172.21.15.195:0/1317570164 wait complete. 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 Processor -- start 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- start start 2024-02-20T00:31:28.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.711+0000 7fed5e48a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed58106ca0 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed57fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed57fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47218/0 (socket says 172.21.15.195:47218) 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed57fff700 1 -- 172.21.15.195:0/1377842848 learned_addr learned my addr 172.21.15.195:0/1377842848 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed57fff700 1 -- 172.21.15.195:0/1377842848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed48005040 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed57fff700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fed48005010 tx=0x7fed4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed48009320 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed58106ea0 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed48013380 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed48014e50 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.712+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed58107330 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.713+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed480095f0 con 0x7fed580f4740 2024-02-20T00:31:28.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.713+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed58045ad0 con 0x7fed580f4740 2024-02-20T00:31:28.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.713+0000 7fed54ff9700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 0x7fed400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:28.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.713+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed48040fa0 con 0x7fed580f4740 2024-02-20T00:31:28.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.713+0000 7fed577fe700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 0x7fed400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:28.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.714+0000 7fed577fe700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 0x7fed400323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fed4c0096a0 tx=0x7fed4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:28.719 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.718+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed48014420 con 0x7fed580f4740 2024-02-20T00:31:28.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:26.575327+0000 mgr.smithi087.iydsqo (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:28.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:27.407184+0000 mon.smithi087 (mon.0) 215 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:31:28.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:31:27.408425+0000 mgr.smithi087.iydsqo (mgr.14162) 171 : cephadm [INF] Deploying daemon node-exporter.smithi087 on smithi087 2024-02-20T00:31:28.925 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.924+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed581081e0 con 0x7fed580f4740 2024-02-20T00:31:28.926 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.925+0000 7fed54ff9700 1 -- 172.21.15.195:0/1377842848 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed48010230 con 0x7fed580f4740 2024-02-20T00:31:28.926 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:28.926 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:28.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 msgr2=0x7fed400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:28.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 0x7fed400323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fed4c0096a0 tx=0x7fed4c006b40).stop 2024-02-20T00:31:28.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 msgr2=0x7fed581067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:28.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fed48005010 tx=0x7fed4800b040).stop 2024-02-20T00:31:28.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 shutdown_connections 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fed4002ff10 0x7fed400323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 --2- 172.21.15.195:0/1377842848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed580f4740 0x7fed581067b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 >> 172.21.15.195:0/1377842848 conn(0x7fed580f0260 msgr2=0x7fed580f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 shutdown_connections 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:28.926+0000 7fed5e48a700 1 -- 172.21.15.195:0/1377842848 wait complete. 2024-02-20T00:31:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:29.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:28.925842+0000 mon.smithi087 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.195:0/1377842848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:30.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:30.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:28.575613+0000 mgr.smithi087.iydsqo (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:32.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.522+0000 7fc5f6085700 1 -- 172.21.15.195:0/3896154562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 msgr2=0x7fc5f00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:32.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.522+0000 7fc5f6085700 1 --2- 172.21.15.195:0/3896154562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f00f7d00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fc5e8009fb0 tx=0x7fc5e8009b30).stop 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- 172.21.15.195:0/3896154562 shutdown_connections 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 --2- 172.21.15.195:0/3896154562 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f00f7d00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- 172.21.15.195:0/3896154562 >> 172.21.15.195:0/3896154562 conn(0x7fc5f00f0120 msgr2=0x7fc5f00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- 172.21.15.195:0/3896154562 shutdown_connections 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- 172.21.15.195:0/3896154562 wait complete. 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 Processor -- start 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- start start 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.523+0000 7fc5f6085700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0108d10 con 0x7fc5f00f78e0 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f5083700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f5083700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47232/0 (socket says 172.21.15.195:47232) 2024-02-20T00:31:32.526 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f5083700 1 -- 172.21.15.195:0/1995567385 learned_addr learned my addr 172.21.15.195:0/1995567385 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:32.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f5083700 1 -- 172.21.15.195:0/1995567385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8005040 con 0x7fc5f00f78e0 2024-02-20T00:31:32.527 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f5083700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc5e80089f0 tx=0x7fc5e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e8009320 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f0108f10 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5e8013380 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e8014e20 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.524+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f01093a0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.525+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f0045ad0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.525+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5e80095f0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.525+0000 7fc5edffb700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 0x7fc5d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:32.528 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.525+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5e8040fb0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.526+0000 7fc5f4882700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 0x7fc5d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:32.529 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.526+0000 7fc5f4882700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 0x7fc5d80323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:32.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.530+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5e806d2b0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.734+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5f010a1f0 con 0x7fc5f00f78e0 2024-02-20T00:31:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.735+0000 7fc5edffb700 1 -- 172.21.15.195:0/1995567385 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5e8014420 con 0x7fc5f00f78e0 2024-02-20T00:31:32.736 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:32.736 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 msgr2=0x7fc5d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 0x7fc5d80323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).stop 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 msgr2=0x7fc5f0108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc5e80089f0 tx=0x7fc5e800b040).stop 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 shutdown_connections 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc5d802ff10 0x7fc5d80323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 --2- 172.21.15.195:0/1995567385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5f00f78e0 0x7fc5f0108820 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 >> 172.21.15.195:0/1995567385 conn(0x7fc5f00f0120 msgr2=0x7fc5f00f0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 shutdown_connections 2024-02-20T00:31:32.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:32.736+0000 7fc5f6085700 1 -- 172.21.15.195:0/1995567385 wait complete. 2024-02-20T00:31:32.739 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:32.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:30.575904+0000 mgr.smithi087.iydsqo (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:33.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:32.735890+0000 mon.smithi087 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.195:0/1995567385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:34.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:34.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:34.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:32.576250+0000 mgr.smithi087.iydsqo (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:36.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 -- 172.21.15.195:0/3907716085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 msgr2=0x7fe26c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:36.242 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 --2- 172.21.15.195:0/3907716085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c0f55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fe25c009fb0 tx=0x7fe25c009b30).stop 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 -- 172.21.15.195:0/3907716085 shutdown_connections 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 --2- 172.21.15.195:0/3907716085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c0f55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 -- 172.21.15.195:0/3907716085 >> 172.21.15.195:0/3907716085 conn(0x7fe26c0f09a0 msgr2=0x7fe26c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 -- 172.21.15.195:0/3907716085 shutdown_connections 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.240+0000 7fe2714a6700 1 -- 172.21.15.195:0/3907716085 wait complete. 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe2714a6700 1 Processor -- start 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe2714a6700 1 -- start start 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe2714a6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:36.243 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe2714a6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c106ad0 con 0x7fe26c0f51a0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe26bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe26bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47258/0 (socket says 172.21.15.195:47258) 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.241+0000 7fe26bfff700 1 -- 172.21.15.195:0/400125345 learned_addr learned my addr 172.21.15.195:0/400125345 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe26bfff700 1 -- 172.21.15.195:0/400125345 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c005040 con 0x7fe26c0f51a0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe26bfff700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe25c009830 tx=0x7fe25c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c009320 con 0x7fe26c0f51a0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c106cd0 con 0x7fe26c0f51a0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe25c013380 con 0x7fe26c0f51a0 2024-02-20T00:31:36.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c014420 con 0x7fe26c0f51a0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.242+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c107160 con 0x7fe26c0f51a0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.243+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe25c01e420 con 0x7fe26c0f51a0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.243+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c045ad0 con 0x7fe26c0f51a0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.243+0000 7fe268ff9700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.243+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe25c040e20 con 0x7fe26c0f51a0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.243+0000 7fe26b7fe700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:36.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.244+0000 7fe26b7fe700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 0x7fe258032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe2540096a0 tx=0x7fe254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:36.249 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.248+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe25c01c970 con 0x7fe26c0f51a0 2024-02-20T00:31:36.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.452+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe26c0fad30 con 0x7fe26c0f51a0 2024-02-20T00:31:36.454 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.453+0000 7fe268ff9700 1 -- 172.21.15.195:0/400125345 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe25c02c2d0 con 0x7fe26c0f51a0 2024-02-20T00:31:36.454 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:36.455 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 msgr2=0x7fe258032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 0x7fe258032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe2540096a0 tx=0x7fe254006b40).stop 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 msgr2=0x7fe26c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe25c009830 tx=0x7fe25c00b040).stop 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 shutdown_connections 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 --2- 172.21.15.195:0/400125345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe26c0f51a0 0x7fe26c1065e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 >> 172.21.15.195:0/400125345 conn(0x7fe26c0f09a0 msgr2=0x7fe26c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 shutdown_connections 2024-02-20T00:31:36.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:36.454+0000 7fe2714a6700 1 -- 172.21.15.195:0/400125345 wait complete. 2024-02-20T00:31:36.457 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:36.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:34.576614+0000 mgr.smithi087.iydsqo (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:37.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:36.453844+0000 mon.smithi087 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.195:0/400125345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:38.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:38.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:36.576883+0000 mgr.smithi087.iydsqo (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:39.987 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4205847713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 msgr2=0x7fcd400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4205847713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd400f55c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fcd30009fb0 tx=0x7fcd30009b30).stop 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4205847713 shutdown_connections 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4205847713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd400f55c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4205847713 >> 172.21.15.195:0/4205847713 conn(0x7fcd400f09a0 msgr2=0x7fcd400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4205847713 shutdown_connections 2024-02-20T00:31:39.988 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.986+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4205847713 wait complete. 2024-02-20T00:31:39.989 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3ffff700 1 Processor -- start 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3ffff700 1 -- start start 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3ffff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401059c0 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:51888/0 (socket says 172.21.15.195:51888) 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.987+0000 7fcd3effd700 1 -- 172.21.15.195:0/4188728729 learned_addr learned my addr 172.21.15.195:0/4188728729 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd3effd700 1 -- 172.21.15.195:0/4188728729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd30005040 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd3effd700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fcd30009830 tx=0x7fcd3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd30009320 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd40105bc0 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd30013380 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd30014420 con 0x7fcd400f51a0 2024-02-20T00:31:39.990 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.988+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd40106050 con 0x7fcd400f51a0 2024-02-20T00:31:39.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd3001e420 con 0x7fcd400f51a0 2024-02-20T00:31:39.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd40045ad0 con 0x7fcd400f51a0 2024-02-20T00:31:39.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd277fe700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 0x7fcd28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:39.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd30040e30 con 0x7fcd400f51a0 2024-02-20T00:31:39.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd3e7fc700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 0x7fcd28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:39.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.989+0000 7fcd3e7fc700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 0x7fcd28032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fcd340096a0 tx=0x7fcd34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:39.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:39.994+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd3001cb10 con 0x7fcd400f51a0 2024-02-20T00:31:40.200 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.198+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd400fad30 con 0x7fcd400f51a0 2024-02-20T00:31:40.201 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.199+0000 7fcd277fe700 1 -- 172.21.15.195:0/4188728729 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd30004490 con 0x7fcd400f51a0 2024-02-20T00:31:40.201 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:40.201 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:40.202 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.200+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 msgr2=0x7fcd28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:40.202 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.200+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 0x7fcd28032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fcd340096a0 tx=0x7fcd34006b40).stop 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 msgr2=0x7fcd401054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fcd30009830 tx=0x7fcd3000b040).stop 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 shutdown_connections 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcd2802ff60 0x7fcd28032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 --2- 172.21.15.195:0/4188728729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcd400f51a0 0x7fcd401054d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 >> 172.21.15.195:0/4188728729 conn(0x7fcd400f09a0 msgr2=0x7fcd400f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 shutdown_connections 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:40.201+0000 7fcd3ffff700 1 -- 172.21.15.195:0/4188728729 wait complete. 2024-02-20T00:31:40.203 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:38.577229+0000 mgr.smithi087.iydsqo (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:40.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:40.200356+0000 mon.smithi087 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.195:0/4188728729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:41.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:41.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:42.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:40.577583+0000 mgr.smithi087.iydsqo (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 -- 172.21.15.195:0/290006782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 msgr2=0x7f80840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 --2- 172.21.15.195:0/290006782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f80840f90f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f8078009fb0 tx=0x7f8078009b30).stop 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 -- 172.21.15.195:0/290006782 shutdown_connections 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 --2- 172.21.15.195:0/290006782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f80840f90f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 -- 172.21.15.195:0/290006782 >> 172.21.15.195:0/290006782 conn(0x7f80840f0b70 msgr2=0x7f80840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 -- 172.21.15.195:0/290006782 shutdown_connections 2024-02-20T00:31:43.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.964+0000 7f808a729700 1 -- 172.21.15.195:0/290006782 wait complete. 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f808a729700 1 Processor -- start 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f808a729700 1 -- start start 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f808a729700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f808a729700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084106c70 con 0x7f8084060280 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f8083fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f8083fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:51906/0 (socket says 172.21.15.195:51906) 2024-02-20T00:31:43.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.965+0000 7f8083fff700 1 -- 172.21.15.195:0/373293118 learned_addr learned my addr 172.21.15.195:0/373293118 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:43.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f8083fff700 1 -- 172.21.15.195:0/373293118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078005040 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f8083fff700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f8078009830 tx=0x7f807800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078009320 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8084106e70 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8078013380 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078014420 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.966+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8084107300 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.967+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f807801e420 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.967+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8084045ad0 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.967+0000 7f8080ff9700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 0x7f806c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.967+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8078040ee0 con 0x7f8084060280 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.967+0000 7f80837fe700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 0x7f806c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:43.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.968+0000 7f80837fe700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 0x7f806c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f80740096a0 tx=0x7f8074006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:43.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:43.972+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f807801c970 con 0x7f8084060280 2024-02-20T00:31:44.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.177+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80840fc090 con 0x7f8084060280 2024-02-20T00:31:44.180 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.178+0000 7f8080ff9700 1 -- 172.21.15.195:0/373293118 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f807802c2d0 con 0x7f8084060280 2024-02-20T00:31:44.180 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:44.180 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 msgr2=0x7f806c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 0x7f806c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f80740096a0 tx=0x7f8074006b40).stop 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 msgr2=0x7f8084106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f8078009830 tx=0x7f807800b040).stop 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 shutdown_connections 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f806c02ff60 0x7f806c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 --2- 172.21.15.195:0/373293118 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8084060280 0x7f8084106780 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 >> 172.21.15.195:0/373293118 conn(0x7f80840f0b70 msgr2=0x7f80840f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 shutdown_connections 2024-02-20T00:31:44.182 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:44.180+0000 7f808a729700 1 -- 172.21.15.195:0/373293118 wait complete. 2024-02-20T00:31:44.183 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:44.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:42.577914+0000 mgr.smithi087.iydsqo (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:44.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:44.179473+0000 mon.smithi087 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.195:0/373293118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:45.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:45.831 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:46.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:44.578265+0000 mgr.smithi087.iydsqo (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:47.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 -- 172.21.15.195:0/3202092147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 msgr2=0x7fea540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 --2- 172.21.15.195:0/3202092147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea540f55c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fea44009fb0 tx=0x7fea44009b30).stop 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 -- 172.21.15.195:0/3202092147 shutdown_connections 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 --2- 172.21.15.195:0/3202092147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea540f55c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 -- 172.21.15.195:0/3202092147 >> 172.21.15.195:0/3202092147 conn(0x7fea540f09a0 msgr2=0x7fea540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 -- 172.21.15.195:0/3202092147 shutdown_connections 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.736+0000 7fea5359e700 1 -- 172.21.15.195:0/3202092147 wait complete. 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5359e700 1 Processor -- start 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5359e700 1 -- start start 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5359e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:47.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5359e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea54060280 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:54364/0 (socket says 172.21.15.195:54364) 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5259c700 1 -- 172.21.15.195:0/1977639297 learned_addr learned my addr 172.21.15.195:0/1977639297 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.737+0000 7fea5259c700 1 -- 172.21.15.195:0/1977639297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44005040 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea5259c700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fea44009830 tx=0x7fea4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea44009320 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea54060480 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea44013380 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea44014420 con 0x7fea540f51a0 2024-02-20T00:31:47.741 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.738+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea541099c0 con 0x7fea540f51a0 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea4401e420 con 0x7fea540f51a0 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea54045ad0 con 0x7fea540f51a0 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea42ffd700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 0x7fea3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea44040ea0 con 0x7fea540f51a0 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea51d9b700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 0x7fea3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:47.742 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.739+0000 7fea51d9b700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 0x7fea3c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fea480096a0 tx=0x7fea48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:47.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.743+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea4401c970 con 0x7fea540f51a0 2024-02-20T00:31:47.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.948+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea540fad30 con 0x7fea540f51a0 2024-02-20T00:31:47.951 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.949+0000 7fea42ffd700 1 -- 172.21.15.195:0/1977639297 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea4402c2d0 con 0x7fea540f51a0 2024-02-20T00:31:47.951 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:47.951 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:47.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 msgr2=0x7fea3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 0x7fea3c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fea480096a0 tx=0x7fea48006b40).stop 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 msgr2=0x7fea54109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fea44009830 tx=0x7fea4400b040).stop 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 shutdown_connections 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea3c02ff60 0x7fea3c032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 --2- 172.21.15.195:0/1977639297 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea540f51a0 0x7fea54109290 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 >> 172.21.15.195:0/1977639297 conn(0x7fea540f09a0 msgr2=0x7fea540f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 shutdown_connections 2024-02-20T00:31:47.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:47.951+0000 7fea5359e700 1 -- 172.21.15.195:0/1977639297 wait complete. 2024-02-20T00:31:47.954 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:46.578622+0000 mgr.smithi087.iydsqo (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:47.950330+0000 mon.smithi087 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.195:0/1977639297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:49.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:49.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:50.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:48.578991+0000 mgr.smithi087.iydsqo (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 -- 172.21.15.195:0/3895624289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 msgr2=0x7ff6200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 --2- 172.21.15.195:0/3895624289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6200f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff610009fb0 tx=0x7ff610009b30).stop 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 -- 172.21.15.195:0/3895624289 shutdown_connections 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 --2- 172.21.15.195:0/3895624289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6200f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 -- 172.21.15.195:0/3895624289 >> 172.21.15.195:0/3895624289 conn(0x7ff6200f0b70 msgr2=0x7ff6200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 -- 172.21.15.195:0/3895624289 shutdown_connections 2024-02-20T00:31:51.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 -- 172.21.15.195:0/3895624289 wait complete. 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.582+0000 7ff625f95700 1 Processor -- start 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff625f95700 1 -- start start 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff625f95700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff625f95700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff620060240 con 0x7ff6200f5410 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:54372/0 (socket says 172.21.15.195:54372) 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff61f7fe700 1 -- 172.21.15.195:0/3809262915 learned_addr learned my addr 172.21.15.195:0/3809262915 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.583+0000 7ff61f7fe700 1 -- 172.21.15.195:0/3809262915 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610005040 con 0x7ff6200f5410 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff61f7fe700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff610005010 tx=0x7ff61000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff610009320 con 0x7ff6200f5410 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff620060440 con 0x7ff6200f5410 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff610013380 con 0x7ff6200f5410 2024-02-20T00:31:51.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff610014e50 con 0x7ff6200f5410 2024-02-20T00:31:51.588 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.584+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff620109b90 con 0x7ff6200f5410 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.585+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6100263a0 con 0x7ff6200f5410 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.585+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff620045ad0 con 0x7ff6200f5410 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.585+0000 7ff617fff700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 0x7ff60c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.585+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff610040e40 con 0x7ff6200f5410 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.585+0000 7ff61effd700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 0x7ff60c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:51.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.586+0000 7ff61effd700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 0x7ff60c0323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff6080096a0 tx=0x7ff608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:51.591 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.590+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff61001ddc0 con 0x7ff6200f5410 2024-02-20T00:31:51.795 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.793+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff62010a3b0 con 0x7ff6200f5410 2024-02-20T00:31:51.796 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.794+0000 7ff617fff700 1 -- 172.21.15.195:0/3809262915 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff61001d9d0 con 0x7ff6200f5410 2024-02-20T00:31:51.796 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:51.796 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:51.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 msgr2=0x7ff60c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:51.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 0x7ff60c0323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff6080096a0 tx=0x7ff608006b40).stop 2024-02-20T00:31:51.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 msgr2=0x7ff6201094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff610005010 tx=0x7ff61000b040).stop 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 shutdown_connections 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff60c02ff10 0x7ff60c0323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 --2- 172.21.15.195:0/3809262915 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6200f5410 0x7ff6201094e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 >> 172.21.15.195:0/3809262915 conn(0x7ff6200f0b70 msgr2=0x7ff6200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.795+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 shutdown_connections 2024-02-20T00:31:51.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:51.796+0000 7ff625f95700 1 -- 172.21.15.195:0/3809262915 wait complete. 2024-02-20T00:31:51.799 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:52.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:50.579356+0000 mgr.smithi087.iydsqo (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:52.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:51.795186+0000 mon.smithi087 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.195:0/3809262915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:53.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:53.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:54.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:52.579704+0000 mgr.smithi087.iydsqo (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:55.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 -- 172.21.15.195:0/3724580362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 msgr2=0x7fa7440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:55.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 --2- 172.21.15.195:0/3724580362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa7440f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fa734009fb0 tx=0x7fa734009b30).stop 2024-02-20T00:31:55.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 -- 172.21.15.195:0/3724580362 shutdown_connections 2024-02-20T00:31:55.321 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 --2- 172.21.15.195:0/3724580362 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa7440f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 -- 172.21.15.195:0/3724580362 >> 172.21.15.195:0/3724580362 conn(0x7fa7440f09a0 msgr2=0x7fa7440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 -- 172.21.15.195:0/3724580362 shutdown_connections 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 -- 172.21.15.195:0/3724580362 wait complete. 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.318+0000 7fa74a23f700 1 Processor -- start 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74a23f700 1 -- start start 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74a23f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74a23f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa744106b40 con 0x7fa7440f51a0 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74923d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74923d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:54380/0 (socket says 172.21.15.195:54380) 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74923d700 1 -- 172.21.15.195:0/2964855312 learned_addr learned my addr 172.21.15.195:0/2964855312 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74923d700 1 -- 172.21.15.195:0/2964855312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734005040 con 0x7fa7440f51a0 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74923d700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa734009f80 tx=0x7fa73400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7340095f0 con 0x7fa7440f51a0 2024-02-20T00:31:55.322 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa734007e90 con 0x7fa7440f51a0 2024-02-20T00:31:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa744106d40 con 0x7fa7440f51a0 2024-02-20T00:31:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.319+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa734014d80 con 0x7fa7440f51a0 2024-02-20T00:31:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.320+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7441071d0 con 0x7fa7440f51a0 2024-02-20T00:31:55.323 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.320+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa734025070 con 0x7fa7440f51a0 2024-02-20T00:31:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.320+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744045ad0 con 0x7fa7440f51a0 2024-02-20T00:31:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.321+0000 7fa739ffb700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 0x7fa730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.321+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa734040830 con 0x7fa7440f51a0 2024-02-20T00:31:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.321+0000 7fa748a3c700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 0x7fa730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:55.324 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.321+0000 7fa748a3c700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 0x7fa730032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa7400096a0 tx=0x7fa740006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:55.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.325+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa73400f4e0 con 0x7fa7440f51a0 2024-02-20T00:31:55.532 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.530+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa744108080 con 0x7fa7440f51a0 2024-02-20T00:31:55.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.531+0000 7fa739ffb700 1 -- 172.21.15.195:0/2964855312 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa73400f300 con 0x7fa7440f51a0 2024-02-20T00:31:55.533 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:55.533 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:55.534 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.532+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 msgr2=0x7fa730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.532+0000 7fa74a23f700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 0x7fa730032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa7400096a0 tx=0x7fa740006b40).stop 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.532+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 msgr2=0x7fa744106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.532+0000 7fa74a23f700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa734009f80 tx=0x7fa73400b040).stop 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 shutdown_connections 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa73002ff60 0x7fa730032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 --2- 172.21.15.195:0/2964855312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7440f51a0 0x7fa744106650 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 >> 172.21.15.195:0/2964855312 conn(0x7fa7440f09a0 msgr2=0x7fa74405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 shutdown_connections 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:55.533+0000 7fa74a23f700 1 -- 172.21.15.195:0/2964855312 wait complete. 2024-02-20T00:31:55.535 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:54.580002+0000 mgr.smithi087.iydsqo (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:55.532306+0000 mon.smithi087 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.195:0/2964855312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:31:57.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:31:57.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:31:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:56.669798+0000 mon.smithi087 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:31:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:56.670583+0000 mon.smithi087 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:31:58.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:56.580288+0000 mgr.smithi087.iydsqo (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:31:59.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 -- 172.21.15.195:0/910891064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 msgr2=0x7f0ae00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/910891064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae00f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f0ac8009fb0 tx=0x7f0ac8009b30).stop 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 -- 172.21.15.195:0/910891064 shutdown_connections 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/910891064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae00f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 -- 172.21.15.195:0/910891064 >> 172.21.15.195:0/910891064 conn(0x7f0ae00f1c70 msgr2=0x7f0ae00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 -- 172.21.15.195:0/910891064 shutdown_connections 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.025+0000 7f0ae4b36700 1 -- 172.21.15.195:0/910891064 wait complete. 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ae4b36700 1 Processor -- start 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ae4b36700 1 -- start start 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ae4b36700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ae4b36700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae0069e20 con 0x7f0ae00f6510 2024-02-20T00:31:59.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:50668/0 (socket says 172.21.15.195:50668) 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.026+0000 7f0ade59c700 1 -- 172.21.15.195:0/2930595131 learned_addr learned my addr 172.21.15.195:0/2930595131 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ade59c700 1 -- 172.21.15.195:0/2930595131 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8005040 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ade59c700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0ac8005010 tx=0x7f0ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8009320 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae0069fc0 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ac8013380 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8014e50 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.027+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae006a450 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.028+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ac8014420 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.028+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae0059100 con 0x7f0ae00f6510 2024-02-20T00:31:59.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.028+0000 7f0ad6ffd700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 0x7f0acc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:31:59.032 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.028+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ac8040f70 con 0x7f0ae00f6510 2024-02-20T00:31:59.032 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.028+0000 7f0addd9b700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 0x7f0acc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:31:59.032 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.029+0000 7f0addd9b700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 0x7f0acc0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0ad00096a0 tx=0x7f0ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:31:59.034 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.033+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ac80095f0 con 0x7f0ae00f6510 2024-02-20T00:31:59.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.243+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ae0045ad0 con 0x7f0ae00f6510 2024-02-20T00:31:59.245 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.243+0000 7f0ad6ffd700 1 -- 172.21.15.195:0/2930595131 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ac801db70 con 0x7f0ae00f6510 2024-02-20T00:31:59.245 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:31:59.245 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:31:59.246 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 msgr2=0x7f0acc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 0x7f0acc0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f0ad00096a0 tx=0x7f0ad0006b40).stop 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 msgr2=0x7f0ae006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0ac8005010 tx=0x7f0ac800b040).stop 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 shutdown_connections 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f0acc02ff10 0x7f0acc0323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 --2- 172.21.15.195:0/2930595131 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ae00f6510 0x7f0ae006b9a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 >> 172.21.15.195:0/2930595131 conn(0x7f0ae00f1c70 msgr2=0x7f0ae00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 shutdown_connections 2024-02-20T00:31:59.247 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:31:59.245+0000 7f0ae4b36700 1 -- 172.21.15.195:0/2930595131 wait complete. 2024-02-20T00:31:59.248 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:31:59.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:31:59 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:31:59.244585+0000 mon.smithi087 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.195:0/2930595131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:31:58.580682+0000 mgr.smithi087.iydsqo (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:00.894 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:00.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:02.702 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 -- 172.21.15.195:0/1142009514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 msgr2=0x7f2b140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:02.811 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 --2- 172.21.15.195:0/1142009514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b140f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2b04009fb0 tx=0x7f2b04009b30).stop 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 -- 172.21.15.195:0/1142009514 shutdown_connections 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 --2- 172.21.15.195:0/1142009514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b140f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 -- 172.21.15.195:0/1142009514 >> 172.21.15.195:0/1142009514 conn(0x7f2b140f0b70 msgr2=0x7f2b140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 -- 172.21.15.195:0/1142009514 shutdown_connections 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.700+0000 7f2b195c5700 1 -- 172.21.15.195:0/1142009514 wait complete. 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b195c5700 1 Processor -- start 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b195c5700 1 -- start start 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b195c5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b195c5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14060220 con 0x7f2b140f5410 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b12ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b12ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:50680/0 (socket says 172.21.15.195:50680) 2024-02-20T00:32:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.701+0000 7f2b12ffd700 1 -- 172.21.15.195:0/3240439757 learned_addr learned my addr 172.21.15.195:0/3240439757 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2b12ffd700 1 -- 172.21.15.195:0/3240439757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04005040 con 0x7f2b140f5410 2024-02-20T00:32:02.813 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:00.581001+0000 mgr.smithi087.iydsqo (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2b12ffd700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2b04005010 tx=0x7f2b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b04009320 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b14060420 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b04013380 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b04014e50 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.702+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14109bb0 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b040263a0 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b14059100 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2afb7fe700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 0x7f2afc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b04041290 con 0x7f2b140f5410 2024-02-20T00:32:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2b127fc700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 0x7f2afc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.703+0000 7f2b127fc700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 0x7f2afc0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2b080096a0 tx=0x7f2b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.707+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b040126f0 con 0x7f2b140f5410 2024-02-20T00:32:02.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.914+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b14045ad0 con 0x7f2b140f5410 2024-02-20T00:32:02.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.915+0000 7f2afb7fe700 1 -- 172.21.15.195:0/3240439757 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b04012330 con 0x7f2b140f5410 2024-02-20T00:32:02.916 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:02.916 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:02.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.916+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 msgr2=0x7f2afc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:02.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.916+0000 7f2b195c5700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 0x7f2afc0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2b080096a0 tx=0x7f2b08006b40).stop 2024-02-20T00:32:02.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.916+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 msgr2=0x7f2b14109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:02.917 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.916+0000 7f2b195c5700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2b04005010 tx=0x7f2b0400b040).stop 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 shutdown_connections 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2afc02ff10 0x7f2afc0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 --2- 172.21.15.195:0/3240439757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b140f5410 0x7f2b14109500 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 >> 172.21.15.195:0/3240439757 conn(0x7f2b140f0b70 msgr2=0x7f2b140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 shutdown_connections 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:02.917+0000 7f2b195c5700 1 -- 172.21.15.195:0/3240439757 wait complete. 2024-02-20T00:32:02.918 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:03.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:03 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:02.915172+0000 mon.smithi087 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.195:0/3240439757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:04.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:04.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:04.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:02.581258+0000 mgr.smithi087.iydsqo (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:06.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 -- 172.21.15.195:0/1096559367 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 msgr2=0x7fc6640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:06.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 --2- 172.21.15.195:0/1096559367 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6640f55c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fc660009fb0 tx=0x7fc660009b30).stop 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 -- 172.21.15.195:0/1096559367 shutdown_connections 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 --2- 172.21.15.195:0/1096559367 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6640f55c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 -- 172.21.15.195:0/1096559367 >> 172.21.15.195:0/1096559367 conn(0x7fc6640f09a0 msgr2=0x7fc6640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 -- 172.21.15.195:0/1096559367 shutdown_connections 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.306+0000 7fc66acd4700 1 -- 172.21.15.195:0/1096559367 wait complete. 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc66acd4700 1 Processor -- start 2024-02-20T00:32:06.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc66acd4700 1 -- start start 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc66acd4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc66acd4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc664060220 con 0x7fc6640f51a0 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc669cd2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc669cd2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53506/0 (socket says 172.21.15.195:53506) 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.307+0000 7fc669cd2700 1 -- 172.21.15.195:0/637386126 learned_addr learned my addr 172.21.15.195:0/637386126 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc669cd2700 1 -- 172.21.15.195:0/637386126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc660005040 con 0x7fc6640f51a0 2024-02-20T00:32:06.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc669cd2700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc6600089f0 tx=0x7fc66000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc660009320 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc660013380 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc664060420 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc660014e20 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.308+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6641099a0 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc660014420 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc664059100 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc65a7fc700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 0x7fc6500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc660040ff0 con 0x7fc6640f51a0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc6694d1700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 0x7fc6500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:06.311 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.309+0000 7fc6694d1700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 0x7fc6500323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc6540096a0 tx=0x7fc654006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:06.314 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.314+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6600095f0 con 0x7fc6640f51a0 2024-02-20T00:32:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.521+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc664045ad0 con 0x7fc6640f51a0 2024-02-20T00:32:06.522 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.522+0000 7fc65a7fc700 1 -- 172.21.15.195:0/637386126 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc66001db70 con 0x7fc6640f51a0 2024-02-20T00:32:06.523 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:06.523 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 msgr2=0x7fc6500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 0x7fc6500323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc6540096a0 tx=0x7fc654006b40).stop 2024-02-20T00:32:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 msgr2=0x7fc6641092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:06.524 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fc6600089f0 tx=0x7fc66000b040).stop 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 shutdown_connections 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc65002ff10 0x7fc6500323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 --2- 172.21.15.195:0/637386126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc6640f51a0 0x7fc6641092f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 >> 172.21.15.195:0/637386126 conn(0x7fc6640f09a0 msgr2=0x7fc6640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 shutdown_connections 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:06.523+0000 7fc66acd4700 1 -- 172.21.15.195:0/637386126 wait complete. 2024-02-20T00:32:06.525 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:06.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:04.581620+0000 mgr.smithi087.iydsqo (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:07.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:06.521946+0000 mon.smithi087 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.195:0/637386126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:08.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:08.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:08.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:06.581986+0000 mgr.smithi087.iydsqo (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 -- 172.21.15.195:0/4150310501 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 msgr2=0x7f42c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 --2- 172.21.15.195:0/4150310501 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c00f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f42b0009fb0 tx=0x7f42b0009b30).stop 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 -- 172.21.15.195:0/4150310501 shutdown_connections 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 --2- 172.21.15.195:0/4150310501 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c00f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 -- 172.21.15.195:0/4150310501 >> 172.21.15.195:0/4150310501 conn(0x7f42c00f0b70 msgr2=0x7f42c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 -- 172.21.15.195:0/4150310501 shutdown_connections 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.179+0000 7f42c82bc700 1 -- 172.21.15.195:0/4150310501 wait complete. 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c82bc700 1 Processor -- start 2024-02-20T00:32:10.183 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c82bc700 1 -- start start 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c82bc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c82bc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0060280 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c6058700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c6058700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53510/0 (socket says 172.21.15.195:53510) 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.180+0000 7f42c6058700 1 -- 172.21.15.195:0/2781235040 learned_addr learned my addr 172.21.15.195:0/2781235040 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42c6058700 1 -- 172.21.15.195:0/2781235040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0005040 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42c6058700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f42b0005010 tx=0x7f42b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b0009320 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c0060480 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42b0013380 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b0014e50 con 0x7f42c00f5410 2024-02-20T00:32:10.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.181+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c0109bd0 con 0x7f42c00f5410 2024-02-20T00:32:10.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.182+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42b00095f0 con 0x7f42c00f5410 2024-02-20T00:32:10.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.182+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0045ad0 con 0x7f42c00f5410 2024-02-20T00:32:10.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.182+0000 7f42b6ffd700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 0x7f42ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:10.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.182+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42b0040fa0 con 0x7f42c00f5410 2024-02-20T00:32:10.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.182+0000 7f42c5857700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 0x7f42ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:10.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.183+0000 7f42c5857700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 0x7f42ac0323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f42bc0096a0 tx=0x7f42bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:10.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.187+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42b0014420 con 0x7f42c00f5410 2024-02-20T00:32:10.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.392+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42c00fafa0 con 0x7f42c00f5410 2024-02-20T00:32:10.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.393+0000 7f42b6ffd700 1 -- 172.21.15.195:0/2781235040 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42b0010230 con 0x7f42c00f5410 2024-02-20T00:32:10.395 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:10.395 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:10.396 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 msgr2=0x7f42ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 0x7f42ac0323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f42bc0096a0 tx=0x7f42bc006b40).stop 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 msgr2=0x7f42c01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f42b0005010 tx=0x7f42b000b040).stop 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 shutdown_connections 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f42ac02ff10 0x7f42ac0323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 --2- 172.21.15.195:0/2781235040 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42c00f5410 0x7f42c01094a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 >> 172.21.15.195:0/2781235040 conn(0x7f42c00f0b70 msgr2=0x7f42c00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 shutdown_connections 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:10.395+0000 7f42c82bc700 1 -- 172.21.15.195:0/2781235040 wait complete. 2024-02-20T00:32:10.397 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:10.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:08.582275+0000 mgr.smithi087.iydsqo (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:10.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:10.394108+0000 mon.smithi087 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.195:0/2781235040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:12.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:12.043 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:12.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:10.582558+0000 mgr.smithi087.iydsqo (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:14.002 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/1748038227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 msgr2=0x7fdeac0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/1748038227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac0f4b60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fdea0009fb0 tx=0x7fdea0009b30).stop 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/1748038227 shutdown_connections 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/1748038227 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac0f4b60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/1748038227 >> 172.21.15.195:0/1748038227 conn(0x7fdeac0f0260 msgr2=0x7fdeac0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/1748038227 shutdown_connections 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.993+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/1748038227 wait complete. 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeb0d9c700 1 Processor -- start 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeb0d9c700 1 -- start start 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeb0d9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeb0d9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac106ca0 con 0x7fdeac0f4740 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeaa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:14.003 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeaa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53522/0 (socket says 172.21.15.195:53522) 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeaa59c700 1 -- 172.21.15.195:0/403593141 learned_addr learned my addr 172.21.15.195:0/403593141 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.994+0000 7fdeaa59c700 1 -- 172.21.15.195:0/403593141 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0005040 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fdeaa59c700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fdea0005010 tx=0x7fdea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea0009320 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac106ea0 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdea0013380 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea0014e50 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.995+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac107330 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac045ad0 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdea00095f0 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fde9affd700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 0x7fde940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdea0040fa0 con 0x7fdeac0f4740 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fdea9d9b700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 0x7fde940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:14.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:13.996+0000 7fdea9d9b700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 0x7fde940323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fde9c0096a0 tx=0x7fde9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:14.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.000+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdea0014420 con 0x7fdeac0f4740 2024-02-20T00:32:14.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.206+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdeac1081e0 con 0x7fdeac0f4740 2024-02-20T00:32:14.208 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.207+0000 7fde9affd700 1 -- 172.21.15.195:0/403593141 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdea0010230 con 0x7fdeac0f4740 2024-02-20T00:32:14.209 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:14.209 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 msgr2=0x7fde940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 0x7fde940323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fde9c0096a0 tx=0x7fde9c006b40).stop 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 msgr2=0x7fdeac1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fdea0005010 tx=0x7fdea000b040).stop 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 shutdown_connections 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fde9402ff10 0x7fde940323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 --2- 172.21.15.195:0/403593141 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f4740 0x7fdeac1067b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 >> 172.21.15.195:0/403593141 conn(0x7fdeac0f0260 msgr2=0x7fdeac0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 shutdown_connections 2024-02-20T00:32:14.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:14.208+0000 7fdeb0d9c700 1 -- 172.21.15.195:0/403593141 wait complete. 2024-02-20T00:32:14.211 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:14.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:12.582878+0000 mgr.smithi087.iydsqo (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:14.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:14.207831+0000 mon.smithi087 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.195:0/403593141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:15.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:15.807 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:16.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:14.583204+0000 mgr.smithi087.iydsqo (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- 172.21.15.195:0/782099683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 msgr2=0x7fe71c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 --2- 172.21.15.195:0/782099683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c0f8f80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fe70c009fb0 tx=0x7fe70c009b30).stop 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- 172.21.15.195:0/782099683 shutdown_connections 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 --2- 172.21.15.195:0/782099683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c0f8f80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- 172.21.15.195:0/782099683 >> 172.21.15.195:0/782099683 conn(0x7fe71c0f09a0 msgr2=0x7fe71c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- 172.21.15.195:0/782099683 shutdown_connections 2024-02-20T00:32:17.758 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- 172.21.15.195:0/782099683 wait complete. 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 Processor -- start 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- start start 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.755+0000 7fe72113c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c106b00 con 0x7fe71c060180 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe71bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe71bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49414/0 (socket says 172.21.15.195:49414) 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe71bfff700 1 -- 172.21.15.195:0/442903534 learned_addr learned my addr 172.21.15.195:0/442903534 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe71bfff700 1 -- 172.21.15.195:0/442903534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c005040 con 0x7fe71c060180 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe71bfff700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fe70c005010 tx=0x7fe70c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe70c009320 con 0x7fe71c060180 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71c106d00 con 0x7fe71c060180 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe70c013380 con 0x7fe71c060180 2024-02-20T00:32:17.759 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe70c014e50 con 0x7fe71c060180 2024-02-20T00:32:17.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.756+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c107190 con 0x7fe71c060180 2024-02-20T00:32:17.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.757+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe70c014420 con 0x7fe71c060180 2024-02-20T00:32:17.760 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.757+0000 7fe718ff9700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 0x7fe7080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:17.761 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.757+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71c045ad0 con 0x7fe71c060180 2024-02-20T00:32:17.761 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.757+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe70c040f40 con 0x7fe71c060180 2024-02-20T00:32:17.761 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.758+0000 7fe71b7fe700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 0x7fe7080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:17.761 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.758+0000 7fe71b7fe700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 0x7fe7080323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe7040096a0 tx=0x7fe704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:17.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.762+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe70c0095f0 con 0x7fe71c060180 2024-02-20T00:32:17.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.968+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe71c108040 con 0x7fe71c060180 2024-02-20T00:32:17.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.969+0000 7fe718ff9700 1 -- 172.21.15.195:0/442903534 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe70c01db70 con 0x7fe71c060180 2024-02-20T00:32:17.971 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:17.971 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:17.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 msgr2=0x7fe7080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:17.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 0x7fe7080323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe7040096a0 tx=0x7fe704006b40).stop 2024-02-20T00:32:17.972 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 msgr2=0x7fe71c106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fe70c005010 tx=0x7fe70c00b040).stop 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 shutdown_connections 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe70802ff10 0x7fe7080323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 --2- 172.21.15.195:0/442903534 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe71c060180 0x7fe71c106610 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 >> 172.21.15.195:0/442903534 conn(0x7fe71c0f09a0 msgr2=0x7fe71c0f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 shutdown_connections 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:17.971+0000 7fe72113c700 1 -- 172.21.15.195:0/442903534 wait complete. 2024-02-20T00:32:17.973 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:18.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:16.583511+0000 mgr.smithi087.iydsqo (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:18.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:17.970125+0000 mon.smithi087 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.195:0/442903534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:19.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:19.741 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:18.583765+0000 mgr.smithi087.iydsqo (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:21.568 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 -- 172.21.15.195:0/2941507023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 msgr2=0x7f449c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 --2- 172.21.15.195:0/2941507023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c0f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f4490009fb0 tx=0x7f4490009b30).stop 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 -- 172.21.15.195:0/2941507023 shutdown_connections 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 --2- 172.21.15.195:0/2941507023 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c0f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 -- 172.21.15.195:0/2941507023 >> 172.21.15.195:0/2941507023 conn(0x7f449c0f1c70 msgr2=0x7f449c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 -- 172.21.15.195:0/2941507023 shutdown_connections 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.566+0000 7f44a323a700 1 -- 172.21.15.195:0/2941507023 wait complete. 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a323a700 1 Processor -- start 2024-02-20T00:32:21.569 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a323a700 1 -- start start 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a323a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a323a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c108ec0 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a0fd6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a0fd6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49432/0 (socket says 172.21.15.195:49432) 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.567+0000 7f44a0fd6700 1 -- 172.21.15.195:0/3179769489 learned_addr learned my addr 172.21.15.195:0/3179769489 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44a0fd6700 1 -- 172.21.15.195:0/3179769489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490005040 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44a0fd6700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4490005010 tx=0x7f449000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490009320 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c1090c0 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4490013380 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490014e50 con 0x7f449c0f6510 2024-02-20T00:32:21.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.568+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c109550 con 0x7f449c0f6510 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.569+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4490014420 con 0x7f449c0f6510 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.569+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c045ad0 con 0x7f449c0f6510 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.569+0000 7f44997fa700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.569+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4490040f40 con 0x7f449c0f6510 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.569+0000 7f449bfff700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:21.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.570+0000 7f449bfff700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 0x7f448c0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f44880096a0 tx=0x7f4488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:21.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.574+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44900095f0 con 0x7f449c0f6510 2024-02-20T00:32:21.780 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.779+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f449c10a400 con 0x7f449c0f6510 2024-02-20T00:32:21.781 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.779+0000 7f44997fa700 1 -- 172.21.15.195:0/3179769489 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f449001db70 con 0x7f449c0f6510 2024-02-20T00:32:21.782 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:21.782 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:21.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 msgr2=0x7f448c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:21.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 0x7f448c0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f44880096a0 tx=0x7f4488006b40).stop 2024-02-20T00:32:21.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 msgr2=0x7f449c1089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:21.783 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4490005010 tx=0x7f449000b040).stop 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 shutdown_connections 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 --2- 172.21.15.195:0/3179769489 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f449c0f6510 0x7f449c1089d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 >> 172.21.15.195:0/3179769489 conn(0x7f449c0f1c70 msgr2=0x7f449c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 shutdown_connections 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:21.781+0000 7f44a323a700 1 -- 172.21.15.195:0/3179769489 wait complete. 2024-02-20T00:32:21.784 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:20.584096+0000 mgr.smithi087.iydsqo (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:22.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:21.780724+0000 mon.smithi087 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.195:0/3179769489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:23.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:23.524 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:24.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:22.584461+0000 mgr.smithi087.iydsqo (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:25.360 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 -- 172.21.15.195:0/3258992341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 msgr2=0x7fadbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 --2- 172.21.15.195:0/3258992341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc0f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fadac009fb0 tx=0x7fadac009b30).stop 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 -- 172.21.15.195:0/3258992341 shutdown_connections 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 --2- 172.21.15.195:0/3258992341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc0f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 -- 172.21.15.195:0/3258992341 >> 172.21.15.195:0/3258992341 conn(0x7fadbc0f0b70 msgr2=0x7fadbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 -- 172.21.15.195:0/3258992341 shutdown_connections 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.358+0000 7fadc3b32700 1 -- 172.21.15.195:0/3258992341 wait complete. 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc3b32700 1 Processor -- start 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc3b32700 1 -- start start 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc3b32700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:25.361 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc3b32700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc105c00 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc18ce700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc18ce700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:49442/0 (socket says 172.21.15.195:49442) 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.359+0000 7fadc18ce700 1 -- 172.21.15.195:0/636508523 learned_addr learned my addr 172.21.15.195:0/636508523 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadc18ce700 1 -- 172.21.15.195:0/636508523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac005040 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadc18ce700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fadac005010 tx=0x7fadac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadac009320 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc105e00 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadac013380 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadac014e50 con 0x7fadbc0f5410 2024-02-20T00:32:25.363 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.360+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc106290 con 0x7fadbc0f5410 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fadac0095f0 con 0x7fadbc0f5410 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadbc045ad0 con 0x7fadbc0f5410 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadb27fc700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 0x7fada80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fadac040e90 con 0x7fadbc0f5410 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadc10cd700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 0x7fada80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:25.364 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.361+0000 7fadc10cd700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 0x7fada80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fadb80096a0 tx=0x7fadb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:25.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.365+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadac014420 con 0x7fadbc0f5410 2024-02-20T00:32:25.571 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.570+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadbc107140 con 0x7fadbc0f5410 2024-02-20T00:32:25.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.571+0000 7fadb27fc700 1 -- 172.21.15.195:0/636508523 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fadac010230 con 0x7fadbc0f5410 2024-02-20T00:32:25.572 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:25.572 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 msgr2=0x7fada80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 0x7fada80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fadb80096a0 tx=0x7fadb8006b40).stop 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 msgr2=0x7fadbc105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fadac005010 tx=0x7fadac00b040).stop 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 shutdown_connections 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fada802ff10 0x7fada80323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:25.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 --2- 172.21.15.195:0/636508523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fadbc0f5410 0x7fadbc105710 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:25.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 >> 172.21.15.195:0/636508523 conn(0x7fadbc0f0b70 msgr2=0x7fadbc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:25.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 shutdown_connections 2024-02-20T00:32:25.575 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:25.572+0000 7fadc3b32700 1 -- 172.21.15.195:0/636508523 wait complete. 2024-02-20T00:32:25.575 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:26.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:24.584851+0000 mgr.smithi087.iydsqo (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:26.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:25.571911+0000 mon.smithi087 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.195:0/636508523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:27.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:27.209 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:28.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:26.585130+0000 mgr.smithi087.iydsqo (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:29.121 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 -- 172.21.15.195:0/3972992013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c0f5410 msgr2=0x7f550c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:29.121 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 --2- 172.21.15.195:0/3972992013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c0f5410 0x7f550c0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f5500009fb0 tx=0x7f5500009b30).stop 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 -- 172.21.15.195:0/3972992013 shutdown_connections 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 --2- 172.21.15.195:0/3972992013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c0f5410 0x7f550c0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 -- 172.21.15.195:0/3972992013 >> 172.21.15.195:0/3972992013 conn(0x7f550c0f0b70 msgr2=0x7f550c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 -- 172.21.15.195:0/3972992013 shutdown_connections 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.118+0000 7f5511d5d700 1 -- 172.21.15.195:0/3972992013 wait complete. 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f5511d5d700 1 Processor -- start 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f5511d5d700 1 -- start start 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f5511d5d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f5511d5d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c107210 con 0x7f550c106900 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f550b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f550b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41050/0 (socket says 172.21.15.195:41050) 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f550b7fe700 1 -- 172.21.15.195:0/83187834 learned_addr learned my addr 172.21.15.195:0/83187834 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.119+0000 7f550b7fe700 1 -- 172.21.15.195:0/83187834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5500005040 con 0x7f550c106900 2024-02-20T00:32:29.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f550b7fe700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f5500009f80 tx=0x7f5500009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:29.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5500013380 con 0x7f550c106900 2024-02-20T00:32:29.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c107410 con 0x7f550c106900 2024-02-20T00:32:29.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5500013790 con 0x7f550c106900 2024-02-20T00:32:29.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5500014420 con 0x7f550c106900 2024-02-20T00:32:29.123 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.120+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c107f60 con 0x7f550c106900 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5500025060 con 0x7f550c106900 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c059100 con 0x7f550c106900 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f54f3fff700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 0x7f54f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5500040e00 con 0x7f550c106900 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f550affd700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 0x7f54f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:29.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.121+0000 7f550affd700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 0x7f54f4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:29.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.125+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f550001c950 con 0x7f550c106900 2024-02-20T00:32:29.334 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.333+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f550c1086b0 con 0x7f550c106900 2024-02-20T00:32:29.335 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.333+0000 7f54f3fff700 1 -- 172.21.15.195:0/83187834 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f550002d2d0 con 0x7f550c106900 2024-02-20T00:32:29.335 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:29.335 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:29.336 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 msgr2=0x7f54f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:29.336 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 0x7f54f4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).stop 2024-02-20T00:32:29.336 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 msgr2=0x7f550c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f5500009f80 tx=0x7f5500009b30).stop 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 shutdown_connections 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f54f402ff60 0x7f54f4032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 --2- 172.21.15.195:0/83187834 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f550c106900 0x7f550c106d20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 >> 172.21.15.195:0/83187834 conn(0x7f550c0f0b70 msgr2=0x7f550c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 shutdown_connections 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:29.335+0000 7f5511d5d700 1 -- 172.21.15.195:0/83187834 wait complete. 2024-02-20T00:32:29.337 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:29.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:29.334471+0000 mon.smithi087 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.195:0/83187834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:28.585403+0000 mgr.smithi087.iydsqo (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:31.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:31.032 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:32.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:30.585659+0000 mgr.smithi087.iydsqo (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:33.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 -- 172.21.15.195:0/1267580449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 msgr2=0x7fda780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:33.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 --2- 172.21.15.195:0/1267580449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fda68009fb0 tx=0x7fda68009b30).stop 2024-02-20T00:32:33.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 -- 172.21.15.195:0/1267580449 shutdown_connections 2024-02-20T00:32:33.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 --2- 172.21.15.195:0/1267580449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:33.005 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 -- 172.21.15.195:0/1267580449 >> 172.21.15.195:0/1267580449 conn(0x7fda780f0b70 msgr2=0x7fda780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 -- 172.21.15.195:0/1267580449 shutdown_connections 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.003+0000 7fda7fa80700 1 -- 172.21.15.195:0/1267580449 wait complete. 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7fa80700 1 Processor -- start 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7fa80700 1 -- start start 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7fa80700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:33.006 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7fa80700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda780fb960 con 0x7fda780f5410 2024-02-20T00:32:33.007 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7d81c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:33.007 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7d81c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41058/0 (socket says 172.21.15.195:41058) 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7d81c700 1 -- 172.21.15.195:0/2283976077 learned_addr learned my addr 172.21.15.195:0/2283976077 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7d81c700 1 -- 172.21.15.195:0/2283976077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68005040 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.004+0000 7fda7d81c700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fda68005010 tx=0x7fda6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.005+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda68009320 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.005+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda780fbb00 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.005+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda68013380 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.005+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda68014e50 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.005+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda780fbf90 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda68014420 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda78059100 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda6e7fc700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 0x7fda640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda68040e30 con 0x7fda780f5410 2024-02-20T00:32:33.008 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda7d01b700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 0x7fda640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:33.009 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.006+0000 7fda7d01b700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 0x7fda640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fda740096a0 tx=0x7fda74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:33.012 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.010+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda680095f0 con 0x7fda780f5410 2024-02-20T00:32:33.218 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.217+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda780fafa0 con 0x7fda780f5410 2024-02-20T00:32:33.219 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.218+0000 7fda6e7fc700 1 -- 172.21.15.195:0/2283976077 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda6801db70 con 0x7fda780f5410 2024-02-20T00:32:33.219 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:33.219 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:33.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 msgr2=0x7fda640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:33.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 0x7fda640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fda740096a0 tx=0x7fda74006b40).stop 2024-02-20T00:32:33.222 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 msgr2=0x7fda780fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fda68005010 tx=0x7fda6800b040).stop 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 shutdown_connections 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fda6402ff10 0x7fda640323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 --2- 172.21.15.195:0/2283976077 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fda780f5410 0x7fda780fd4e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 >> 172.21.15.195:0/2283976077 conn(0x7fda780f0b70 msgr2=0x7fda780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 shutdown_connections 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:33.219+0000 7fda7fa80700 1 -- 172.21.15.195:0/2283976077 wait complete. 2024-02-20T00:32:33.223 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:32.585994+0000 mgr.smithi087.iydsqo (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:34.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:33.218704+0000 mon.smithi087 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.195:0/2283976077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:34.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:34.889 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:36.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:34.586374+0000 mgr.smithi087.iydsqo (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:36.847 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 -- 172.21.15.195:0/2663238306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 msgr2=0x7f17940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 --2- 172.21.15.195:0/2663238306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17940f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1788009fb0 tx=0x7f1788009b30).stop 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 -- 172.21.15.195:0/2663238306 shutdown_connections 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 --2- 172.21.15.195:0/2663238306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17940f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 -- 172.21.15.195:0/2663238306 >> 172.21.15.195:0/2663238306 conn(0x7f17940f1c70 msgr2=0x7f17940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 -- 172.21.15.195:0/2663238306 shutdown_connections 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.844+0000 7f1799076700 1 -- 172.21.15.195:0/2663238306 wait complete. 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1799076700 1 Processor -- start 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1799076700 1 -- start start 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1799076700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1799076700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1794106cd0 con 0x7f17940f6510 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1792d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:36.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1792d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39448/0 (socket says 172.21.15.195:39448) 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.845+0000 7f1792d9d700 1 -- 172.21.15.195:0/4289482124 learned_addr learned my addr 172.21.15.195:0/4289482124 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f1792d9d700 1 -- 172.21.15.195:0/4289482124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788005040 con 0x7f17940f6510 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f1792d9d700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1788005010 tx=0x7f178800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1788009320 con 0x7f17940f6510 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1794106ed0 con 0x7f17940f6510 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1788013380 con 0x7f17940f6510 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1788014e50 con 0x7f17940f6510 2024-02-20T00:32:36.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.846+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1794107360 con 0x7f17940f6510 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17880263a0 con 0x7f17940f6510 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f177b7fe700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 0x7f177c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1788040ef0 con 0x7f17940f6510 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1794045ad0 con 0x7f17940f6510 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f179259c700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 0x7f177c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:36.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.847+0000 7f179259c700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 0x7f177c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f17840096a0 tx=0x7f1784006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:36.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:36.852+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1788012920 con 0x7f17940f6510 2024-02-20T00:32:37.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.056+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17940fc190 con 0x7f17940f6510 2024-02-20T00:32:37.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.057+0000 7f177b7fe700 1 -- 172.21.15.195:0/4289482124 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f178802b020 con 0x7f17940f6510 2024-02-20T00:32:37.059 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:37.059 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:37.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 msgr2=0x7f177c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 0x7f177c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f17840096a0 tx=0x7f1784006b40).stop 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 msgr2=0x7f17941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1788005010 tx=0x7f178800b040).stop 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 shutdown_connections 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f177c02ff10 0x7f177c0323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 --2- 172.21.15.195:0/4289482124 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f17940f6510 0x7f17941067e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 >> 172.21.15.195:0/4289482124 conn(0x7f17940f1c70 msgr2=0x7f17940f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 shutdown_connections 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:37.058+0000 7f1799076700 1 -- 172.21.15.195:0/4289482124 wait complete. 2024-02-20T00:32:37.061 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:37.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:37.058074+0000 mon.smithi087 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.195:0/4289482124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:38.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:36.586736+0000 mgr.smithi087.iydsqo (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:38.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:38.550 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:40.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 -- 172.21.15.195:0/3708315250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 msgr2=0x7f205c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:40.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 --2- 172.21.15.195:0/3708315250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c0f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f204c009fb0 tx=0x7f204c009b30).stop 2024-02-20T00:32:40.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 -- 172.21.15.195:0/3708315250 shutdown_connections 2024-02-20T00:32:40.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 --2- 172.21.15.195:0/3708315250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c0f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:40.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 -- 172.21.15.195:0/3708315250 >> 172.21.15.195:0/3708315250 conn(0x7f205c0f1c70 msgr2=0x7f205c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:40.414 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 -- 172.21.15.195:0/3708315250 shutdown_connections 2024-02-20T00:32:40.414 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.411+0000 7f20616fd700 1 -- 172.21.15.195:0/3708315250 wait complete. 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.412+0000 7f20616fd700 1 Processor -- start 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.412+0000 7f20616fd700 1 -- start start 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.412+0000 7f20616fd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.412+0000 7f20616fd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c106cd0 con 0x7f205c0f6510 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f205affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f205affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39460/0 (socket says 172.21.15.195:39460) 2024-02-20T00:32:40.415 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f205affd700 1 -- 172.21.15.195:0/2121215494 learned_addr learned my addr 172.21.15.195:0/2121215494 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f205affd700 1 -- 172.21.15.195:0/2121215494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c005040 con 0x7f205c0f6510 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f205affd700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f204c005010 tx=0x7f204c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f204c009320 con 0x7f205c0f6510 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205c106ed0 con 0x7f205c0f6510 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f204c013380 con 0x7f205c0f6510 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.413+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f204c014e50 con 0x7f205c0f6510 2024-02-20T00:32:40.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.414+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c107360 con 0x7f205c0f6510 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.414+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f204c014420 con 0x7f205c0f6510 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.414+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205c045ad0 con 0x7f205c0f6510 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.414+0000 7f20437fe700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 0x7f20440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.414+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f204c040e30 con 0x7f205c0f6510 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.415+0000 7f205a7fc700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 0x7f20440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:40.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.415+0000 7f205a7fc700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 0x7f20440323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f20500096a0 tx=0x7f2050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:40.421 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.419+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f204c0095f0 con 0x7f205c0f6510 2024-02-20T00:32:40.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:38.587029+0000 mgr.smithi087.iydsqo (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:40.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.623+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f205c108210 con 0x7f205c0f6510 2024-02-20T00:32:40.625 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.624+0000 7f20437fe700 1 -- 172.21.15.195:0/2121215494 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f204c01db70 con 0x7f205c0f6510 2024-02-20T00:32:40.626 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:40.626 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:40.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 msgr2=0x7f20440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:40.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 0x7f20440323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f20500096a0 tx=0x7f2050006b40).stop 2024-02-20T00:32:40.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 msgr2=0x7f205c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:40.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f204c005010 tx=0x7f204c00b040).stop 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 shutdown_connections 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f204402ff10 0x7f20440323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 --2- 172.21.15.195:0/2121215494 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f205c0f6510 0x7f205c1067e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.625+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 >> 172.21.15.195:0/2121215494 conn(0x7f205c0f1c70 msgr2=0x7f205c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.626+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 shutdown_connections 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:40.626+0000 7f20616fd700 1 -- 172.21.15.195:0/2121215494 wait complete. 2024-02-20T00:32:40.628 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:41.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:40.625267+0000 mon.smithi087 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.195:0/2121215494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:42.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:42.303 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:42.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:40.587323+0000 mgr.smithi087.iydsqo (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:44.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- 172.21.15.195:0/965260368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 msgr2=0x7f3ce80f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 --2- 172.21.15.195:0/965260368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce80f6d10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f3ce4009fb0 tx=0x7f3ce4009b30).stop 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- 172.21.15.195:0/965260368 shutdown_connections 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 --2- 172.21.15.195:0/965260368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce80f6d10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- 172.21.15.195:0/965260368 >> 172.21.15.195:0/965260368 conn(0x7f3ce80f09a0 msgr2=0x7f3ce80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- 172.21.15.195:0/965260368 shutdown_connections 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- 172.21.15.195:0/965260368 wait complete. 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 Processor -- start 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- start start 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.325+0000 7f3ceea52700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce810d0e0 con 0x7f3ce80601e0 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3ceda50700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:44.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3ceda50700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39468/0 (socket says 172.21.15.195:39468) 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3ceda50700 1 -- 172.21.15.195:0/1736633788 learned_addr learned my addr 172.21.15.195:0/1736633788 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3ceda50700 1 -- 172.21.15.195:0/1736633788 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4005040 con 0x7f3ce80601e0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3ceda50700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f3ce4005010 tx=0x7f3ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4009320 con 0x7f3ce80601e0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.326+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ce4013380 con 0x7f3ce80601e0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.327+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4014e50 con 0x7f3ce80601e0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.327+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce810d2e0 con 0x7f3ce80601e0 2024-02-20T00:32:44.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.327+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce810d770 con 0x7f3ce80601e0 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.327+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce40095f0 con 0x7f3ce80601e0 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.327+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce8045ad0 con 0x7f3ce80601e0 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.328+0000 7f3cde7fc700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 0x7f3cd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.328+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce4040f70 con 0x7f3ce80601e0 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.328+0000 7f3ced24f700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 0x7f3cd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:44.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.328+0000 7f3ced24f700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 0x7f3cd40323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3cd80096a0 tx=0x7f3cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:44.334 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.333+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce4014420 con 0x7f3ce80601e0 2024-02-20T00:32:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:42.587613+0000 mgr.smithi087.iydsqo (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:44.540 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.539+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ce810e620 con 0x7f3ce80601e0 2024-02-20T00:32:44.541 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.539+0000 7f3cde7fc700 1 -- 172.21.15.195:0/1736633788 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ce401d980 con 0x7f3ce80601e0 2024-02-20T00:32:44.541 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:44.541 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:44.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 msgr2=0x7f3cd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:44.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 0x7f3cd40323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3cd80096a0 tx=0x7f3cd8006b40).stop 2024-02-20T00:32:44.543 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 msgr2=0x7f3ce810cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f3ce4005010 tx=0x7f3ce400b040).stop 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 shutdown_connections 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3cd402ff10 0x7f3cd40323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 --2- 172.21.15.195:0/1736633788 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ce80601e0 0x7f3ce810cbf0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 >> 172.21.15.195:0/1736633788 conn(0x7f3ce80f09a0 msgr2=0x7f3ce80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 shutdown_connections 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:44.541+0000 7f3ceea52700 1 -- 172.21.15.195:0/1736633788 wait complete. 2024-02-20T00:32:44.544 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:45.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:44.540707+0000 mon.smithi087 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.195:0/1736633788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:46.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:46.219 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:44.587972+0000 mgr.smithi087.iydsqo (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.158+0000 7f670cc33700 1 -- 172.21.15.195:0/37376876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 msgr2=0x7f67080f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.158+0000 7f670cc33700 1 --2- 172.21.15.195:0/37376876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f4b60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f66f0009fb0 tx=0x7f66f0009b30).stop 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.158+0000 7f670cc33700 1 -- 172.21.15.195:0/37376876 shutdown_connections 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.158+0000 7f670cc33700 1 --2- 172.21.15.195:0/37376876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f4b60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.158+0000 7f670cc33700 1 -- 172.21.15.195:0/37376876 >> 172.21.15.195:0/37376876 conn(0x7f67080f0260 msgr2=0x7f67080f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 -- 172.21.15.195:0/37376876 shutdown_connections 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 -- 172.21.15.195:0/37376876 wait complete. 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 Processor -- start 2024-02-20T00:32:48.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 -- start start 2024-02-20T00:32:48.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:48.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670cc33700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080f5770 con 0x7f67080f4740 2024-02-20T00:32:48.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:48.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55488/0 (socket says 172.21.15.195:55488) 2024-02-20T00:32:48.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.159+0000 7f670659c700 1 -- 172.21.15.195:0/1453886135 learned_addr learned my addr 172.21.15.195:0/1453886135 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:48.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f670659c700 1 -- 172.21.15.195:0/1453886135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0005040 con 0x7f67080f4740 2024-02-20T00:32:48.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f670659c700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f66f0005010 tx=0x7f66f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:48.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0009320 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66f0013380 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67080f5910 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0014e50 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.160+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67080f5da0 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.161+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708059100 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.161+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66f00095f0 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.161+0000 7f66feffd700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 0x7f66f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.161+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66f0040e90 con 0x7f67080f4740 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.161+0000 7f6705d9b700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 0x7f66f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:48.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.162+0000 7f6705d9b700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 0x7f66f40323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f66f80096a0 tx=0x7f66f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:48.167 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.166+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66f0014420 con 0x7f67080f4740 2024-02-20T00:32:48.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.369+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67080fc080 con 0x7f67080f4740 2024-02-20T00:32:48.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.370+0000 7f66feffd700 1 -- 172.21.15.195:0/1453886135 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66f0010230 con 0x7f67080f4740 2024-02-20T00:32:48.372 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:48.372 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 msgr2=0x7f66f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 0x7f66f40323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f66f80096a0 tx=0x7f66f8006b40).stop 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 msgr2=0x7f67080f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f66f0005010 tx=0x7f66f000b040).stop 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 shutdown_connections 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f66f402ff10 0x7f66f40323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 --2- 172.21.15.195:0/1453886135 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67080f4740 0x7f67080f72f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 >> 172.21.15.195:0/1453886135 conn(0x7f67080f0260 msgr2=0x7f67080f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 shutdown_connections 2024-02-20T00:32:48.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:48.372+0000 7f670cc33700 1 -- 172.21.15.195:0/1453886135 wait complete. 2024-02-20T00:32:48.375 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:48.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:46.588274+0000 mgr.smithi087.iydsqo (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:49.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:48.371388+0000 mon.smithi087 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.195:0/1453886135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:50.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:50.010 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:50.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:48.588641+0000 mgr.smithi087.iydsqo (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:51.891 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- 172.21.15.195:0/3074084675 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 msgr2=0x7f22ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 --2- 172.21.15.195:0/3074084675 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec0f66c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f22e8009fb0 tx=0x7f22e8009b30).stop 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- 172.21.15.195:0/3074084675 shutdown_connections 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 --2- 172.21.15.195:0/3074084675 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec0f66c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- 172.21.15.195:0/3074084675 >> 172.21.15.195:0/3074084675 conn(0x7f22ec0f1aa0 msgr2=0x7f22ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- 172.21.15.195:0/3074084675 shutdown_connections 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- 172.21.15.195:0/3074084675 wait complete. 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 Processor -- start 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.889+0000 7f22f2c79700 1 -- start start 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f2c79700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f2c79700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec069e20 con 0x7f22ec0f62a0 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f1c77700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f1c77700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55512/0 (socket says 172.21.15.195:55512) 2024-02-20T00:32:51.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f1c77700 1 -- 172.21.15.195:0/1567049949 learned_addr learned my addr 172.21.15.195:0/1567049949 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f1c77700 1 -- 172.21.15.195:0/1567049949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e8005040 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f1c77700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f22e80089f0 tx=0x7f22e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22e8009320 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.890+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec069fc0 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22e8013380 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22e8014e20 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec06a450 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22e80095f0 con 0x7f22ec0f62a0 2024-02-20T00:32:51.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec059100 con 0x7f22ec0f62a0 2024-02-20T00:32:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.891+0000 7f22e27fc700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 0x7f22d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.892+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22e8040f10 con 0x7f22ec0f62a0 2024-02-20T00:32:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.892+0000 7f22f1476700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 0x7f22d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:51.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.892+0000 7f22f1476700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 0x7f22d80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f22dc0096a0 tx=0x7f22dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:51.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:51.897+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22e8014420 con 0x7f22ec0f62a0 2024-02-20T00:32:52.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.103+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22ec0fbe30 con 0x7f22ec0f62a0 2024-02-20T00:32:52.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.103+0000 7f22e27fc700 1 -- 172.21.15.195:0/1567049949 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22e801d980 con 0x7f22ec0f62a0 2024-02-20T00:32:52.105 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:52.105 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:52.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 msgr2=0x7f22d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 0x7f22d80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f22dc0096a0 tx=0x7f22dc006b40).stop 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 msgr2=0x7f22ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f22e80089f0 tx=0x7f22e800b040).stop 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 shutdown_connections 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f22d802ff10 0x7f22d80323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 --2- 172.21.15.195:0/1567049949 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f22ec0f62a0 0x7f22ec06b9a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 >> 172.21.15.195:0/1567049949 conn(0x7f22ec0f1aa0 msgr2=0x7f22ec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 shutdown_connections 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:52.105+0000 7f22f2c79700 1 -- 172.21.15.195:0/1567049949 wait complete. 2024-02-20T00:32:52.107 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:52.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:50.589046+0000 mgr.smithi087.iydsqo (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:52.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:52.104475+0000 mon.smithi087 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.195:0/1567049949' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:53.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:53.788 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:52.589369+0000 mgr.smithi087.iydsqo (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 -- 172.21.15.195:0/4109769412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 msgr2=0x7f72900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 --2- 172.21.15.195:0/4109769412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72900f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f7280009fb0 tx=0x7f7280009b30).stop 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 -- 172.21.15.195:0/4109769412 shutdown_connections 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 --2- 172.21.15.195:0/4109769412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72900f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 -- 172.21.15.195:0/4109769412 >> 172.21.15.195:0/4109769412 conn(0x7f72900f0b70 msgr2=0x7f72900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 -- 172.21.15.195:0/4109769412 shutdown_connections 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.735+0000 7f7297bce700 1 -- 172.21.15.195:0/4109769412 wait complete. 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f7297bce700 1 Processor -- start 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f7297bce700 1 -- start start 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f7297bce700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f7297bce700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290106ca0 con 0x7f72900f5410 2024-02-20T00:32:55.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f729596a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f729596a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55518/0 (socket says 172.21.15.195:55518) 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.736+0000 7f729596a700 1 -- 172.21.15.195:0/3461458254 learned_addr learned my addr 172.21.15.195:0/3461458254 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f729596a700 1 -- 172.21.15.195:0/3461458254 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280005040 con 0x7f72900f5410 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f729596a700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7280005010 tx=0x7f728000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7280009320 con 0x7f72900f5410 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7290106ea0 con 0x7f72900f5410 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7280013380 con 0x7f72900f5410 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7280014e50 con 0x7f72900f5410 2024-02-20T00:32:55.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.737+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7290107330 con 0x7f72900f5410 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290045ad0 con 0x7f72900f5410 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72800095f0 con 0x7f72900f5410 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f72867fc700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 0x7f727c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7280040e90 con 0x7f72900f5410 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f7295169700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 0x7f727c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:55.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.738+0000 7f7295169700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 0x7f727c0323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f728c0096a0 tx=0x7f728c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:55.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.742+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7280014420 con 0x7f72900f5410 2024-02-20T00:32:55.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.946+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72901081e0 con 0x7f72900f5410 2024-02-20T00:32:55.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.947+0000 7f72867fc700 1 -- 172.21.15.195:0/3461458254 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7280010230 con 0x7f72900f5410 2024-02-20T00:32:55.947 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:32:55.948 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 msgr2=0x7f727c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 0x7f727c0323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f728c0096a0 tx=0x7f728c006b40).stop 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 msgr2=0x7f72901067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7280005010 tx=0x7f728000b040).stop 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 shutdown_connections 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f727c02ff10 0x7f727c0323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 --2- 172.21.15.195:0/3461458254 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72900f5410 0x7f72901067b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:55.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 >> 172.21.15.195:0/3461458254 conn(0x7f72900f0b70 msgr2=0x7f729005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:55.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 shutdown_connections 2024-02-20T00:32:55.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:55.948+0000 7f7297bce700 1 -- 172.21.15.195:0/3461458254 wait complete. 2024-02-20T00:32:55.950 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:32:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:54.589704+0000 mgr.smithi087.iydsqo (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:55.947074+0000 mon.smithi087 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.195:0/3461458254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:32:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:56.670918+0000 mon.smithi087 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:32:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:32:56.671609+0000 mon.smithi087 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:32:57.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:32:57.693 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:32:58.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:32:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:56.590081+0000 mgr.smithi087.iydsqo (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 -- 172.21.15.195:0/3984287355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 msgr2=0x7f9b180f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3984287355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f4ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9b0c009fb0 tx=0x7f9b0c009b30).stop 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 -- 172.21.15.195:0/3984287355 shutdown_connections 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3984287355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f4ca0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 -- 172.21.15.195:0/3984287355 >> 172.21.15.195:0/3984287355 conn(0x7f9b180f02e0 msgr2=0x7f9b180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 -- 172.21.15.195:0/3984287355 shutdown_connections 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.809+0000 7f9b1d857700 1 -- 172.21.15.195:0/3984287355 wait complete. 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.810+0000 7f9b1d857700 1 Processor -- start 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.810+0000 7f9b1d857700 1 -- start start 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b1d857700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:59.813 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b1d857700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180f58f0 con 0x7f9b180f4880 2024-02-20T00:32:59.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:59.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b16ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:43126/0 (socket says 172.21.15.195:43126) 2024-02-20T00:32:59.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b16ffd700 1 -- 172.21.15.195:0/3371442912 learned_addr learned my addr 172.21.15.195:0/3371442912 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:32:59.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b16ffd700 1 -- 172.21.15.195:0/3371442912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c005040 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.811+0000 7f9b16ffd700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9b0c0089f0 tx=0x7f9b0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b0c009320 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b180f5a90 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b0c013380 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b0c014e20 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b180f5f20 con 0x7f9b180f4880 2024-02-20T00:32:59.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b0c014420 con 0x7f9b180f4880 2024-02-20T00:32:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.812+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18059100 con 0x7f9b180f4880 2024-02-20T00:32:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.813+0000 7f9aff7fe700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 0x7f9b000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:32:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.813+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b0c040ed0 con 0x7f9b180f4880 2024-02-20T00:32:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.813+0000 7f9b167fc700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 0x7f9b000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:32:59.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.813+0000 7f9b167fc700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 0x7f9b000323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:32:59.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:32:59.818+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b0c014790 con 0x7f9b180f4880 2024-02-20T00:33:00.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.026+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b18045ad0 con 0x7f9b180f4880 2024-02-20T00:33:00.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.027+0000 7f9aff7fe700 1 -- 172.21.15.195:0/3371442912 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b0c01d7f0 con 0x7f9b180f4880 2024-02-20T00:33:00.029 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:00.029 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 msgr2=0x7f9b000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 0x7f9b000323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).stop 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 msgr2=0x7f9b180f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9b0c0089f0 tx=0x7f9b0c00b040).stop 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 shutdown_connections 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9b0002ff10 0x7f9b000323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:00.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 --2- 172.21.15.195:0/3371442912 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b180f4880 0x7f9b180f7470 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:00.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 >> 172.21.15.195:0/3371442912 conn(0x7f9b180f02e0 msgr2=0x7f9b180f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:00.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 shutdown_connections 2024-02-20T00:33:00.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:00.028+0000 7f9b1d857700 1 -- 172.21.15.195:0/3371442912 wait complete. 2024-02-20T00:33:00.031 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:00.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:32:58.590414+0000 mgr.smithi087.iydsqo (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:00.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:00.027866+0000 mon.smithi087 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.195:0/3371442912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:01.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:01.710 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:02.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:00.590734+0000 mgr.smithi087.iydsqo (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 -- 172.21.15.195:0/3669861816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 msgr2=0x7fefd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 --2- 172.21.15.195:0/3669861816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd80f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fefc8009fb0 tx=0x7fefc8009b30).stop 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 -- 172.21.15.195:0/3669861816 shutdown_connections 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 --2- 172.21.15.195:0/3669861816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd80f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 -- 172.21.15.195:0/3669861816 >> 172.21.15.195:0/3669861816 conn(0x7fefd80f0b70 msgr2=0x7fefd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 -- 172.21.15.195:0/3669861816 shutdown_connections 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.568+0000 7fefe0393700 1 -- 172.21.15.195:0/3669861816 wait complete. 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefe0393700 1 Processor -- start 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefe0393700 1 -- start start 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefe0393700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefe0393700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd8106cd0 con 0x7fefd80f5410 2024-02-20T00:33:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefde12f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefde12f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:43134/0 (socket says 172.21.15.195:43134) 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefde12f700 1 -- 172.21.15.195:0/2912627375 learned_addr learned my addr 172.21.15.195:0/2912627375 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.569+0000 7fefde12f700 1 -- 172.21.15.195:0/2912627375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc8005040 con 0x7fefd80f5410 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefde12f700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fefc8005010 tx=0x7fefc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefc8009320 con 0x7fefd80f5410 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd8106ed0 con 0x7fefd80f5410 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefc8013380 con 0x7fefd80f5410 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefc8014e50 con 0x7fefd80f5410 2024-02-20T00:33:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.570+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd8107360 con 0x7fefd80f5410 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefc80095f0 con 0x7fefd80f5410 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd8045ad0 con 0x7fefd80f5410 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefceffd700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefc8040fa0 con 0x7fefd80f5410 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefdd92e700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.571+0000 7fefdd92e700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 0x7fefc40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fefd40096a0 tx=0x7fefd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.576+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefc8014420 con 0x7fefd80f5410 2024-02-20T00:33:03.779 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.778+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefd80fafa0 con 0x7fefd80f5410 2024-02-20T00:33:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.779+0000 7fefceffd700 1 -- 172.21.15.195:0/2912627375 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefc8010230 con 0x7fefd80f5410 2024-02-20T00:33:03.780 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:03.780 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 msgr2=0x7fefc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 0x7fefc40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fefd40096a0 tx=0x7fefd4006b40).stop 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 msgr2=0x7fefd81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fefc8005010 tx=0x7fefc800b040).stop 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 shutdown_connections 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 --2- 172.21.15.195:0/2912627375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefd80f5410 0x7fefd81067e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 >> 172.21.15.195:0/2912627375 conn(0x7fefd80f0b70 msgr2=0x7fefd805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 shutdown_connections 2024-02-20T00:33:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:03.780+0000 7fefe0393700 1 -- 172.21.15.195:0/2912627375 wait complete. 2024-02-20T00:33:03.783 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:04.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:02.591013+0000 mgr.smithi087.iydsqo (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:03.779822+0000 mon.smithi087 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.195:0/2912627375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:05.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:05.382 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:06.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:04.591315+0000 mgr.smithi087.iydsqo (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 -- 172.21.15.195:0/4075256819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 msgr2=0x7fb53c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 --2- 172.21.15.195:0/4075256819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c0f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb524009fb0 tx=0x7fb524009b30).stop 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 -- 172.21.15.195:0/4075256819 shutdown_connections 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 --2- 172.21.15.195:0/4075256819 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c0f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 -- 172.21.15.195:0/4075256819 >> 172.21.15.195:0/4075256819 conn(0x7fb53c0f1c70 msgr2=0x7fb53c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 -- 172.21.15.195:0/4075256819 shutdown_connections 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.203+0000 7fb54272f700 1 -- 172.21.15.195:0/4075256819 wait complete. 2024-02-20T00:33:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.204+0000 7fb54272f700 1 Processor -- start 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.204+0000 7fb54272f700 1 -- start start 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.204+0000 7fb54272f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.204+0000 7fb54272f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c106cd0 con 0x7fb53c0f6510 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb53bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb53bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59980/0 (socket says 172.21.15.195:59980) 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb53bfff700 1 -- 172.21.15.195:0/1461872320 learned_addr learned my addr 172.21.15.195:0/1461872320 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb53bfff700 1 -- 172.21.15.195:0/1461872320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb524005040 con 0x7fb53c0f6510 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb53bfff700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb524005010 tx=0x7fb52400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb524009320 con 0x7fb53c0f6510 2024-02-20T00:33:07.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c106ed0 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb524013380 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb524014e50 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.205+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c107360 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.206+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5240263a0 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.206+0000 7fb538ff9700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 0x7fb5280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.206+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb524040f10 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.206+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c045ad0 con 0x7fb53c0f6510 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.206+0000 7fb53b7fe700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 0x7fb5280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:07.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.207+0000 7fb53b7fe700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 0x7fb5280323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb52c0096a0 tx=0x7fb52c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:07.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.211+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb524012920 con 0x7fb53c0f6510 2024-02-20T00:33:07.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.414+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb53c108080 con 0x7fb53c0f6510 2024-02-20T00:33:07.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.415+0000 7fb538ff9700 1 -- 172.21.15.195:0/1461872320 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb52402b020 con 0x7fb53c0f6510 2024-02-20T00:33:07.417 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:07.417 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 msgr2=0x7fb5280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 0x7fb5280323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb52c0096a0 tx=0x7fb52c006b40).stop 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 msgr2=0x7fb53c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb524005010 tx=0x7fb52400b040).stop 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 shutdown_connections 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb52802ff10 0x7fb5280323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 --2- 172.21.15.195:0/1461872320 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb53c0f6510 0x7fb53c1067e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 >> 172.21.15.195:0/1461872320 conn(0x7fb53c0f1c70 msgr2=0x7fb53c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 shutdown_connections 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:07.417+0000 7fb54272f700 1 -- 172.21.15.195:0/1461872320 wait complete. 2024-02-20T00:33:07.419 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:08.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:06.591720+0000 mgr.smithi087.iydsqo (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:08.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:07.416302+0000 mon.smithi087 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.195:0/1461872320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:09.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:09.057 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:10.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:08.592095+0000 mgr.smithi087.iydsqo (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 -- 172.21.15.195:0/636235653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 msgr2=0x7fd9800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 --2- 172.21.15.195:0/636235653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9800f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fd974009fb0 tx=0x7fd974009b30).stop 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 -- 172.21.15.195:0/636235653 shutdown_connections 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 --2- 172.21.15.195:0/636235653 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9800f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 -- 172.21.15.195:0/636235653 >> 172.21.15.195:0/636235653 conn(0x7fd9800f0b70 msgr2=0x7fd9800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:10.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 -- 172.21.15.195:0/636235653 shutdown_connections 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.889+0000 7fd985b65700 1 -- 172.21.15.195:0/636235653 wait complete. 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd985b65700 1 Processor -- start 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd985b65700 1 -- start start 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd985b65700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd985b65700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd980106ca0 con 0x7fd9800f5410 2024-02-20T00:33:10.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd97f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd97f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59990/0 (socket says 172.21.15.195:59990) 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.890+0000 7fd97f7fe700 1 -- 172.21.15.195:0/4003478908 learned_addr learned my addr 172.21.15.195:0/4003478908 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd97f7fe700 1 -- 172.21.15.195:0/4003478908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974005040 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd97f7fe700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd9740089f0 tx=0x7fd97400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd974009320 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd974013380 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd980106ea0 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd97401ed80 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.891+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd980107330 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd97401e420 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd980045ad0 con 0x7fd9800f5410 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd967fff700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:10.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd974040e10 con 0x7fd9800f5410 2024-02-20T00:33:10.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd97effd700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:10.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.892+0000 7fd97effd700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd9700096a0 tx=0x7fd970006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:10.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:10.896+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd97401c930 con 0x7fd9800f5410 2024-02-20T00:33:11.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.101+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9801081e0 con 0x7fd9800f5410 2024-02-20T00:33:11.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.102+0000 7fd967fff700 1 -- 172.21.15.195:0/4003478908 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd97401cb40 con 0x7fd9800f5410 2024-02-20T00:33:11.104 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:11.104 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 msgr2=0x7fd968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 0x7fd968032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd9700096a0 tx=0x7fd970006b40).stop 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 msgr2=0x7fd9801067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd9740089f0 tx=0x7fd97400b040).stop 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 shutdown_connections 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd96802ff60 0x7fd968032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 --2- 172.21.15.195:0/4003478908 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9800f5410 0x7fd9801067b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:11.105 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.103+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 >> 172.21.15.195:0/4003478908 conn(0x7fd9800f0b70 msgr2=0x7fd98005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:11.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.104+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 shutdown_connections 2024-02-20T00:33:11.106 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:11.104+0000 7fd985b65700 1 -- 172.21.15.195:0/4003478908 wait complete. 2024-02-20T00:33:11.106 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:11.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:11.103173+0000 mon.smithi087 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.195:0/4003478908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:12.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:10.592523+0000 mgr.smithi087.iydsqo (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:12.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:12.740 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:12.592984+0000 mgr.smithi087.iydsqo (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:14.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 -- 172.21.15.195:0/569489251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 msgr2=0x7f4b4c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:14.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 --2- 172.21.15.195:0/569489251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f4b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f4b34009fb0 tx=0x7f4b34009b30).stop 2024-02-20T00:33:14.584 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 -- 172.21.15.195:0/569489251 shutdown_connections 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 --2- 172.21.15.195:0/569489251 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f4b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 -- 172.21.15.195:0/569489251 >> 172.21.15.195:0/569489251 conn(0x7f4b4c0f0260 msgr2=0x7f4b4c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 -- 172.21.15.195:0/569489251 shutdown_connections 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.581+0000 7f4b50901700 1 -- 172.21.15.195:0/569489251 wait complete. 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b50901700 1 Processor -- start 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b50901700 1 -- start start 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b50901700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b50901700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0f5770 con 0x7f4b4c0f4740 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b4a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b4a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60006/0 (socket says 172.21.15.195:60006) 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.582+0000 7f4b4a59c700 1 -- 172.21.15.195:0/3172375598 learned_addr learned my addr 172.21.15.195:0/3172375598 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b4a59c700 1 -- 172.21.15.195:0/3172375598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34005040 con 0x7f4b4c0f4740 2024-02-20T00:33:14.585 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b4a59c700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f4b340089f0 tx=0x7f4b3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:14.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b34009320 con 0x7f4b4c0f4740 2024-02-20T00:33:14.586 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c0f5910 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b34013380 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3401ed80 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.583+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c0f5da0 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b3401e420 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c059100 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b42ffd700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 0x7f4b38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b34041090 con 0x7f4b4c0f4740 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b49d9b700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 0x7f4b38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:14.587 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.584+0000 7f4b49d9b700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 0x7f4b38032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4b3c0096a0 tx=0x7f4b3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:14.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.589+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b340095f0 con 0x7f4b4c0f4740 2024-02-20T00:33:14.793 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.791+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b4c0fc080 con 0x7f4b4c0f4740 2024-02-20T00:33:14.794 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.792+0000 7f4b42ffd700 1 -- 172.21.15.195:0/3172375598 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b34004490 con 0x7f4b4c0f4740 2024-02-20T00:33:14.794 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:14.795 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 msgr2=0x7f4b38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 0x7f4b38032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4b3c0096a0 tx=0x7f4b3c006b40).stop 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 msgr2=0x7f4b4c0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f4b340089f0 tx=0x7f4b3400b040).stop 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 shutdown_connections 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4b3802ff60 0x7f4b38032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 --2- 172.21.15.195:0/3172375598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b4c0f4740 0x7f4b4c0f72f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 >> 172.21.15.195:0/3172375598 conn(0x7f4b4c0f0260 msgr2=0x7f4b4c0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:14.797 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 shutdown_connections 2024-02-20T00:33:14.798 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:14.793+0000 7f4b50901700 1 -- 172.21.15.195:0/3172375598 wait complete. 2024-02-20T00:33:14.798 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:15.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:15 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:14.793041+0000 mon.smithi087 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.195:0/3172375598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:14.593334+0000 mgr.smithi087.iydsqo (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:16.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:16.540 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- 172.21.15.195:0/1385195191 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 msgr2=0x7f2da00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/1385195191 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da00f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2d90009fb0 tx=0x7f2d90009b30).stop 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- 172.21.15.195:0/1385195191 shutdown_connections 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/1385195191 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da00f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- 172.21.15.195:0/1385195191 >> 172.21.15.195:0/1385195191 conn(0x7f2da00f09a0 msgr2=0x7f2da00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- 172.21.15.195:0/1385195191 shutdown_connections 2024-02-20T00:33:18.305 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- 172.21.15.195:0/1385195191 wait complete. 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 Processor -- start 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- start start 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.302+0000 7f2da6e7a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da0106ac0 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da5e78700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da5e78700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:32962/0 (socket says 172.21.15.195:32962) 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da5e78700 1 -- 172.21.15.195:0/2859910549 learned_addr learned my addr 172.21.15.195:0/2859910549 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da5e78700 1 -- 172.21.15.195:0/2859910549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90005040 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da5e78700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2d900089f0 tx=0x7f2d9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d90009320 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da0106cc0 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d90013380 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d90014e20 con 0x7f2da00f51a0 2024-02-20T00:33:18.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.303+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da0107150 con 0x7f2da00f51a0 2024-02-20T00:33:18.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.304+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d900095f0 con 0x7f2da00f51a0 2024-02-20T00:33:18.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.304+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da0045ad0 con 0x7f2da00f51a0 2024-02-20T00:33:18.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.304+0000 7f2d967fc700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 0x7f2d8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:18.307 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.304+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d90040e90 con 0x7f2da00f51a0 2024-02-20T00:33:18.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.305+0000 7f2da5677700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 0x7f2d8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:18.308 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.305+0000 7f2da5677700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 0x7f2d8c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2d9c0096a0 tx=0x7f2d9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:18.310 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.309+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d90014420 con 0x7f2da00f51a0 2024-02-20T00:33:18.516 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.514+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2da00fad30 con 0x7f2da00f51a0 2024-02-20T00:33:18.517 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.515+0000 7f2d967fc700 1 -- 172.21.15.195:0/2859910549 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d9001d980 con 0x7f2da00f51a0 2024-02-20T00:33:18.518 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:18.518 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 msgr2=0x7f2d8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 0x7f2d8c0323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2d9c0096a0 tx=0x7f2d9c006b40).stop 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 msgr2=0x7f2da01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2d900089f0 tx=0x7f2d9000b040).stop 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 shutdown_connections 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d8c02ff10 0x7f2d8c0323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 --2- 172.21.15.195:0/2859910549 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2da00f51a0 0x7f2da01065d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:18.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 >> 172.21.15.195:0/2859910549 conn(0x7f2da00f09a0 msgr2=0x7f2da005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:18.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 shutdown_connections 2024-02-20T00:33:18.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:18.517+0000 7f2da6e7a700 1 -- 172.21.15.195:0/2859910549 wait complete. 2024-02-20T00:33:18.520 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:16.593723+0000 mgr.smithi087.iydsqo (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:19.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:19 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:18.516141+0000 mon.smithi087 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.195:0/2859910549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:20.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:20.088 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:20.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:18.594073+0000 mgr.smithi087.iydsqo (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:21.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 -- 172.21.15.195:0/2660515429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 msgr2=0x7f29940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:21.928 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 --2- 172.21.15.195:0/2660515429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29940f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2990009fb0 tx=0x7f2990009b30).stop 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 -- 172.21.15.195:0/2660515429 shutdown_connections 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 --2- 172.21.15.195:0/2660515429 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29940f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 -- 172.21.15.195:0/2660515429 >> 172.21.15.195:0/2660515429 conn(0x7f29940f1c70 msgr2=0x7f29940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 -- 172.21.15.195:0/2660515429 shutdown_connections 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.925+0000 7f299c28e700 1 -- 172.21.15.195:0/2660515429 wait complete. 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299c28e700 1 Processor -- start 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299c28e700 1 -- start start 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299c28e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299c28e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2994108eb0 con 0x7f29940f6510 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299a02a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299a02a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:32988/0 (socket says 172.21.15.195:32988) 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299a02a700 1 -- 172.21.15.195:0/3961499373 learned_addr learned my addr 172.21.15.195:0/3961499373 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299a02a700 1 -- 172.21.15.195:0/3961499373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990005040 con 0x7f29940f6510 2024-02-20T00:33:21.929 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.926+0000 7f299a02a700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2990005010 tx=0x7f299000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2990009320 con 0x7f29940f6510 2024-02-20T00:33:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29941090b0 con 0x7f29940f6510 2024-02-20T00:33:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2990013380 con 0x7f29940f6510 2024-02-20T00:33:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2990014e50 con 0x7f29940f6510 2024-02-20T00:33:21.930 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2994109540 con 0x7f29940f6510 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.927+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29900095f0 con 0x7f29940f6510 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.928+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2994045ad0 con 0x7f29940f6510 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.928+0000 7f298affd700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 0x7f29800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.928+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2990040e70 con 0x7f29940f6510 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.928+0000 7f2999829700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 0x7f29800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:21.931 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.928+0000 7f2999829700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 0x7f29800323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f29840096a0 tx=0x7f2984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:21.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:21.932+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2990014420 con 0x7f29940f6510 2024-02-20T00:33:22.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.136+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f299410a390 con 0x7f29940f6510 2024-02-20T00:33:22.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.137+0000 7f298affd700 1 -- 172.21.15.195:0/3961499373 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2990010230 con 0x7f29940f6510 2024-02-20T00:33:22.139 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:22.139 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 msgr2=0x7f29800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 0x7f29800323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f29840096a0 tx=0x7f2984006b40).stop 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 msgr2=0x7f29941089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f2990005010 tx=0x7f299000b040).stop 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 shutdown_connections 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f298002ff10 0x7f29800323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 --2- 172.21.15.195:0/3961499373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29940f6510 0x7f29941089c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 >> 172.21.15.195:0/3961499373 conn(0x7f29940f1c70 msgr2=0x7f29940f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 shutdown_connections 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:22.139+0000 7f299c28e700 1 -- 172.21.15.195:0/3961499373 wait complete. 2024-02-20T00:33:22.141 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:22.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:20.594421+0000 mgr.smithi087.iydsqo (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:22.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:22.138303+0000 mon.smithi087 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.195:0/3961499373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:23.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:23.743 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:24.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:22.594826+0000 mgr.smithi087.iydsqo (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:25.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 -- 172.21.15.195:0/1463091745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 msgr2=0x7f283c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:25.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 --2- 172.21.15.195:0/1463091745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c0f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f282c009fb0 tx=0x7f282c009b30).stop 2024-02-20T00:33:25.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 -- 172.21.15.195:0/1463091745 shutdown_connections 2024-02-20T00:33:25.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 --2- 172.21.15.195:0/1463091745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c0f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:25.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 -- 172.21.15.195:0/1463091745 >> 172.21.15.195:0/1463091745 conn(0x7f283c0f1c70 msgr2=0x7f283c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:25.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 -- 172.21.15.195:0/1463091745 shutdown_connections 2024-02-20T00:33:25.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.642+0000 7f28442a1700 1 -- 172.21.15.195:0/1463091745 wait complete. 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f28442a1700 1 Processor -- start 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f28442a1700 1 -- start start 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f28442a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f28442a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c069e20 con 0x7f283c0f6510 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f284203d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.643+0000 7f284203d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:33006/0 (socket says 172.21.15.195:33006) 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f284203d700 1 -- 172.21.15.195:0/1296203495 learned_addr learned my addr 172.21.15.195:0/1296203495 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f284203d700 1 -- 172.21.15.195:0/1296203495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c005040 con 0x7f283c0f6510 2024-02-20T00:33:25.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f284203d700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f282c005010 tx=0x7f282c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c009320 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f282c013380 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c069fc0 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f282c014e50 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.644+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c06a450 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.645+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f282c0263a0 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.645+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c059100 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.645+0000 7f2832ffd700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 0x7f28280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.645+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f282c041140 con 0x7f283c0f6510 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.645+0000 7f284183c700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 0x7f28280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:25.649 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.646+0000 7f284183c700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 0x7f28280323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f28380096a0 tx=0x7f2838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:25.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.650+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f282c01cb10 con 0x7f283c0f6510 2024-02-20T00:33:25.857 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.855+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f283c0fc0a0 con 0x7f283c0f6510 2024-02-20T00:33:25.857 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.856+0000 7f2832ffd700 1 -- 172.21.15.195:0/1296203495 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f282c01cb10 con 0x7f283c0f6510 2024-02-20T00:33:25.858 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:25.858 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:25.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 msgr2=0x7f28280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:25.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 0x7f28280323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f28380096a0 tx=0x7f2838006b40).stop 2024-02-20T00:33:25.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 msgr2=0x7f283c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:25.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f282c005010 tx=0x7f282c00b040).stop 2024-02-20T00:33:25.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 shutdown_connections 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f282802ff10 0x7f28280323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 --2- 172.21.15.195:0/1296203495 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f283c0f6510 0x7f283c06b9a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.857+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 >> 172.21.15.195:0/1296203495 conn(0x7f283c0f1c70 msgr2=0x7f283c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.858+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 shutdown_connections 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:25.858+0000 7f28442a1700 1 -- 172.21.15.195:0/1296203495 wait complete. 2024-02-20T00:33:25.860 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:26.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:24.595093+0000 mgr.smithi087.iydsqo (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:26.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:25.857072+0000 mon.smithi087 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.195:0/1296203495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:27.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:27.622 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:28.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:26.595430+0000 mgr.smithi087.iydsqo (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:29.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:33:28.070565+0000 mgr.smithi087.iydsqo (mgr.14162) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi087 ... 2024-02-20T00:33:29.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Verifying port 9100 ... 2024-02-20T00:33:29.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 1 from systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087 2024-02-20T00:33:29.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: systemctl: stderr Job for ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087.service failed because a timeout was exceeded. 2024-02-20T00:33:29.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: systemctl: stderr See "systemctl status ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087.service" and "journalctl -xe" for details. 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ports=daemon_ports) 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T00:33:29.533 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: Failed command: systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: yield (conn, connr) 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: code, '\n'.join(err))) 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi087 ... 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Verifying port 9100 ... 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 1 from systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: systemctl: stderr Job for ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087.service failed because a timeout was exceeded. 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: systemctl: stderr See "systemctl status ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087.service" and "journalctl -xe" for details. 2024-02-20T00:33:29.534 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: ports=daemon_ports) 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T00:33:29.535 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-20T00:33:29.536 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T00:33:29.536 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: Failed command: systemctl start ceph-6575da04-cf86-11ee-95bb-87774f69a715@node-exporter.smithi087 2024-02-20T00:33:29.536 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:33:28.076149+0000 mgr.smithi087.iydsqo (mgr.14162) 233 : cephadm [INF] Deploying daemon prometheus.smithi087 on smithi087 2024-02-20T00:33:29.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 -- 172.21.15.195:0/2338866389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340f5410 msgr2=0x7f83340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 --2- 172.21.15.195:0/2338866389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340f5410 0x7f83340f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8324009fb0 tx=0x7f8324009b30).stop 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 -- 172.21.15.195:0/2338866389 shutdown_connections 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 --2- 172.21.15.195:0/2338866389 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340f5410 0x7f83340f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 -- 172.21.15.195:0/2338866389 >> 172.21.15.195:0/2338866389 conn(0x7f83340f0b70 msgr2=0x7f83340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 -- 172.21.15.195:0/2338866389 shutdown_connections 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 -- 172.21.15.195:0/2338866389 wait complete. 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.589+0000 7f833bf31700 1 Processor -- start 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f833bf31700 1 -- start start 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f833bf31700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f833bf31700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83340fd940 con 0x7f83340fd520 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f8339ccd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f8339ccd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39512/0 (socket says 172.21.15.195:39512) 2024-02-20T00:33:29.593 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f8339ccd700 1 -- 172.21.15.195:0/2356646595 learned_addr learned my addr 172.21.15.195:0/2356646595 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f8339ccd700 1 -- 172.21.15.195:0/2356646595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324005040 con 0x7f83340fd520 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f8339ccd700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f83340f6510 tx=0x7f8324009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.590+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8324013380 con 0x7f83340fd520 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8324013790 con 0x7f83340fd520 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832401ee70 con 0x7f83340fd520 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83340fbed0 con 0x7f83340fd520 2024-02-20T00:33:29.594 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83340fc3c0 con 0x7f83340fd520 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8324025060 con 0x7f83340fd520 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.591+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8334045ad0 con 0x7f83340fd520 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.592+0000 7f832a7fc700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 0x7f8320032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.592+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8324040d40 con 0x7f83340fd520 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.592+0000 7f83394cc700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 0x7f8320032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:29.595 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.592+0000 7f83394cc700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 0x7f8320032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f83300096a0 tx=0x7f8330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:29.598 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.596+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f832401c520 con 0x7f83340fd520 2024-02-20T00:33:29.800 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.798+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f833410cb00 con 0x7f83340fd520 2024-02-20T00:33:29.801 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.799+0000 7f832a7fc700 1 -- 172.21.15.195:0/2356646595 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f832402d2d0 con 0x7f83340fd520 2024-02-20T00:33:29.801 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:29.801 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:29.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 msgr2=0x7f8320032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:29.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 0x7f8320032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f83300096a0 tx=0x7f8330006b40).stop 2024-02-20T00:33:29.802 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 msgr2=0x7f83340fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f83340f6510 tx=0x7f8324009b30).stop 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 shutdown_connections 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f832002ffb0 0x7f8320032470 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 --2- 172.21.15.195:0/2356646595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83340fd520 0x7f83340fb9e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 >> 172.21.15.195:0/2356646595 conn(0x7f83340f0b70 msgr2=0x7f833405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 shutdown_connections 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:29.801+0000 7f833bf31700 1 -- 172.21.15.195:0/2356646595 wait complete. 2024-02-20T00:33:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:30.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:28.595846+0000 mgr.smithi087.iydsqo (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:30.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:29.800267+0000 mon.smithi087 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.195:0/2356646595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:31.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:31.417 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:32.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:30.596140+0000 mgr.smithi087.iydsqo (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.157+0000 7feba3db8700 1 -- 172.21.15.195:0/3510124375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 msgr2=0x7feb9c0f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.157+0000 7feba3db8700 1 --2- 172.21.15.195:0/3510124375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f91b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7feb98009fb0 tx=0x7feb98009b30).stop 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- 172.21.15.195:0/3510124375 shutdown_connections 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 --2- 172.21.15.195:0/3510124375 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f91b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- 172.21.15.195:0/3510124375 >> 172.21.15.195:0/3510124375 conn(0x7feb9c0f0b70 msgr2=0x7feb9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- 172.21.15.195:0/3510124375 shutdown_connections 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- 172.21.15.195:0/3510124375 wait complete. 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 Processor -- start 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- start start 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:33.160 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.158+0000 7feba3db8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0f50b0 con 0x7feb9c0601c0 2024-02-20T00:33:33.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba1b54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:33.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba1b54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:39520/0 (socket says 172.21.15.195:39520) 2024-02-20T00:33:33.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba1b54700 1 -- 172.21.15.195:0/2109762673 learned_addr learned my addr 172.21.15.195:0/2109762673 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:33.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba1b54700 1 -- 172.21.15.195:0/2109762673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98005040 con 0x7feb9c0601c0 2024-02-20T00:33:33.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba1b54700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feb980089f0 tx=0x7feb9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb98009320 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c0f5250 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb98013380 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb98014e20 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.159+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c0f56e0 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.160+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c059100 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.160+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb980095f0 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.160+0000 7feb927fc700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 0x7feb880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.160+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb98040ef0 con 0x7feb9c0601c0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.160+0000 7feba1353700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 0x7feb880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:33.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.161+0000 7feba1353700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 0x7feb880323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feb8c0096a0 tx=0x7feb8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:33.166 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.165+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb98014420 con 0x7feb9c0601c0 2024-02-20T00:33:33.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.369+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb9c045ad0 con 0x7feb9c0601c0 2024-02-20T00:33:33.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.370+0000 7feb927fc700 1 -- 172.21.15.195:0/2109762673 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb98010230 con 0x7feb9c0601c0 2024-02-20T00:33:33.372 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:33.372 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 msgr2=0x7feb880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 0x7feb880323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feb8c0096a0 tx=0x7feb8c006b40).stop 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 msgr2=0x7feb9c0f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feb980089f0 tx=0x7feb9800b040).stop 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 shutdown_connections 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7feb8802ff10 0x7feb880323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:33.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 --2- 172.21.15.195:0/2109762673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7feb9c0601c0 0x7feb9c0f6c30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:33.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 >> 172.21.15.195:0/2109762673 conn(0x7feb9c0f0b70 msgr2=0x7feb9c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:33.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 shutdown_connections 2024-02-20T00:33:33.374 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:33.371+0000 7feba3db8700 1 -- 172.21.15.195:0/2109762673 wait complete. 2024-02-20T00:33:33.374 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:32.596455+0000 mgr.smithi087.iydsqo (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:33.371029+0000 mon.smithi087 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.195:0/2109762673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:34.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:34.997 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:36.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:34.596834+0000 mgr.smithi087.iydsqo (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 -- 172.21.15.195:0/2959152807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 msgr2=0x7f2bec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 --2- 172.21.15.195:0/2959152807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec0f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f2bdc009fb0 tx=0x7f2bdc009b30).stop 2024-02-20T00:33:36.814 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 -- 172.21.15.195:0/2959152807 shutdown_connections 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 --2- 172.21.15.195:0/2959152807 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec0f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 -- 172.21.15.195:0/2959152807 >> 172.21.15.195:0/2959152807 conn(0x7f2bec0f0b70 msgr2=0x7f2bec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 -- 172.21.15.195:0/2959152807 shutdown_connections 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 -- 172.21.15.195:0/2959152807 wait complete. 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.811+0000 7f2bf3562700 1 Processor -- start 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf3562700 1 -- start start 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf3562700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf3562700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec060240 con 0x7f2bec0f5410 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf12fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf12fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:56912/0 (socket says 172.21.15.195:56912) 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf12fe700 1 -- 172.21.15.195:0/193935190 learned_addr learned my addr 172.21.15.195:0/193935190 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf12fe700 1 -- 172.21.15.195:0/193935190 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc005040 con 0x7f2bec0f5410 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2bf12fe700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2bdc005010 tx=0x7f2bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.812+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bdc009320 con 0x7f2bec0f5410 2024-02-20T00:33:36.815 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.813+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec060440 con 0x7f2bec0f5410 2024-02-20T00:33:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.813+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bdc013380 con 0x7f2bec0f5410 2024-02-20T00:33:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.813+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bdc014e50 con 0x7f2bec0f5410 2024-02-20T00:33:36.816 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.813+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec109bc0 con 0x7f2bec0f5410 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec045ad0 con 0x7f2bec0f5410 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bdc0095f0 con 0x7f2bec0f5410 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2be1ffb700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 0x7f2bd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bdc040e90 con 0x7f2bec0f5410 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2bf0afd700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 0x7f2bd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:36.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.814+0000 7f2bf0afd700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 0x7f2bd80323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2be80096a0 tx=0x7f2be8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:36.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:36.818+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bdc014420 con 0x7f2bec0f5410 2024-02-20T00:33:37.026 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.025+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bec10a3d0 con 0x7f2bec0f5410 2024-02-20T00:33:37.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.025+0000 7f2be1ffb700 1 -- 172.21.15.195:0/193935190 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bdc010230 con 0x7f2bec0f5410 2024-02-20T00:33:37.027 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:37.027 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:37.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 msgr2=0x7f2bd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:37.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 0x7f2bd80323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2be80096a0 tx=0x7f2be8006b40).stop 2024-02-20T00:33:37.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 msgr2=0x7f2bec109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2bdc005010 tx=0x7f2bdc00b040).stop 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 shutdown_connections 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2bd802ff10 0x7f2bd80323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 --2- 172.21.15.195:0/193935190 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2bec0f5410 0x7f2bec109510 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 >> 172.21.15.195:0/193935190 conn(0x7f2bec0f0b70 msgr2=0x7f2bec0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 shutdown_connections 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:37.027+0000 7f2bf3562700 1 -- 172.21.15.195:0/193935190 wait complete. 2024-02-20T00:33:37.029 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:37.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:37.026510+0000 mon.smithi087 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.195:0/193935190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:38.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:36.597197+0000 mgr.smithi087.iydsqo (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:38.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:38.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:40.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:38.597565+0000 mgr.smithi087.iydsqo (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:40.559 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 -- 172.21.15.195:0/2964898766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 msgr2=0x7f40480f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 --2- 172.21.15.195:0/2964898766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40480f91b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f4044009fb0 tx=0x7f4044009b30).stop 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 -- 172.21.15.195:0/2964898766 shutdown_connections 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 --2- 172.21.15.195:0/2964898766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40480f91b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 -- 172.21.15.195:0/2964898766 >> 172.21.15.195:0/2964898766 conn(0x7f40480f0b70 msgr2=0x7f40480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 -- 172.21.15.195:0/2964898766 shutdown_connections 2024-02-20T00:33:41.630 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.557+0000 7f404ff6b700 1 -- 172.21.15.195:0/2964898766 wait complete. 2024-02-20T00:33:41.631 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:40.770014+0000 mon.smithi087 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.195:0/1724500677' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404ff6b700 1 Processor -- start 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404ff6b700 1 -- start start 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404ff6b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404ff6b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048106cb0 con 0x7f40480601c0 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404dd07700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:41.634 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404dd07700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:56926/0 (socket says 172.21.15.195:56926) 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.558+0000 7f404dd07700 1 -- 172.21.15.195:0/1724500677 learned_addr learned my addr 172.21.15.195:0/1724500677 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f404dd07700 1 -- 172.21.15.195:0/1724500677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044005040 con 0x7f40480601c0 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f404dd07700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f40440089f0 tx=0x7f404400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4044009320 con 0x7f40480601c0 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4048106eb0 con 0x7f40480601c0 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4044013380 con 0x7f40480601c0 2024-02-20T00:33:41.635 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4044014e20 con 0x7f40480601c0 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.559+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4048107340 con 0x7f40480601c0 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.560+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40440095f0 con 0x7f40480601c0 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.560+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4048045ad0 con 0x7f40480601c0 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.560+0000 7f403e7fc700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 0x7f40340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.560+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4044040e90 con 0x7f40480601c0 2024-02-20T00:33:41.636 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.560+0000 7f404d506700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 0x7f40340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.561+0000 7f404d506700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 0x7f40340323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f40380096a0 tx=0x7f4038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.565+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4044014420 con 0x7f40480601c0 2024-02-20T00:33:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.768+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40481081f0 con 0x7f40480601c0 2024-02-20T00:33:41.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.769+0000 7f403e7fc700 1 -- 172.21.15.195:0/1724500677 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f404401d980 con 0x7f40480601c0 2024-02-20T00:33:41.638 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:41.638 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 msgr2=0x7f40340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 0x7f40340323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f40380096a0 tx=0x7f4038006b40).stop 2024-02-20T00:33:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 msgr2=0x7f40481067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:41.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f40440089f0 tx=0x7f404400b040).stop 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 shutdown_connections 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f403402ff10 0x7f40340323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 --2- 172.21.15.195:0/1724500677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f40480601c0 0x7f40481067c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 >> 172.21.15.195:0/1724500677 conn(0x7f40480f0b70 msgr2=0x7f40480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 shutdown_connections 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:40.770+0000 7f404ff6b700 1 -- 172.21.15.195:0/1724500677 wait complete. 2024-02-20T00:33:41.641 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:42.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:40.597953+0000 mgr.smithi087.iydsqo (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:42.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:42.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:44.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 -- 172.21.15.195:0/1501238036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 msgr2=0x7fcee40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:44.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 --2- 172.21.15.195:0/1501238036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee40f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fced8009fb0 tx=0x7fced8009b30).stop 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 -- 172.21.15.195:0/1501238036 shutdown_connections 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 --2- 172.21.15.195:0/1501238036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee40f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 -- 172.21.15.195:0/1501238036 >> 172.21.15.195:0/1501238036 conn(0x7fcee40f0b70 msgr2=0x7fcee40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 -- 172.21.15.195:0/1501238036 shutdown_connections 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.465+0000 7fceeaa75700 1 -- 172.21.15.195:0/1501238036 wait complete. 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fceeaa75700 1 Processor -- start 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fceeaa75700 1 -- start start 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fceeaa75700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fceeaa75700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4106cc0 con 0x7fcee40f5410 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fcee8811700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fcee8811700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:56950/0 (socket says 172.21.15.195:56950) 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.466+0000 7fcee8811700 1 -- 172.21.15.195:0/750732284 learned_addr learned my addr 172.21.15.195:0/750732284 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:44.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fcee8811700 1 -- 172.21.15.195:0/750732284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8005040 con 0x7fcee40f5410 2024-02-20T00:33:44.469 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fcee8811700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fced8005010 tx=0x7fced800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fced8009320 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4106ec0 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fced8013380 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fced8014e50 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.467+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee4107350 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.468+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fced80095f0 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.468+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4045ad0 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.468+0000 7fcee17fa700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 0x7fced40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.468+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fced8040f10 con 0x7fcee40f5410 2024-02-20T00:33:44.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.468+0000 7fcee3fff700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 0x7fced40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:44.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.469+0000 7fcee3fff700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 0x7fced40323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:44.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.473+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fced8014420 con 0x7fcee40f5410 2024-02-20T00:33:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:42.598259+0000 mgr.smithi087.iydsqo (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.678+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcee4108200 con 0x7fcee40f5410 2024-02-20T00:33:44.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.679+0000 7fcee17fa700 1 -- 172.21.15.195:0/750732284 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fced8010230 con 0x7fcee40f5410 2024-02-20T00:33:44.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:44.680 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 msgr2=0x7fced40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 0x7fced40323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).stop 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 msgr2=0x7fcee41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fced8005010 tx=0x7fced800b040).stop 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 shutdown_connections 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fced402ff10 0x7fced40323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:44.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 --2- 172.21.15.195:0/750732284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcee40f5410 0x7fcee41067d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:44.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.680+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 >> 172.21.15.195:0/750732284 conn(0x7fcee40f0b70 msgr2=0x7fcee405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:44.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.681+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 shutdown_connections 2024-02-20T00:33:44.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:44.681+0000 7fceeaa75700 1 -- 172.21.15.195:0/750732284 wait complete. 2024-02-20T00:33:44.682 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:45.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:44.678987+0000 mon.smithi087 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.195:0/750732284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:46.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:46.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:44.598533+0000 mgr.smithi087.iydsqo (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:48.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- 172.21.15.195:0/2601710129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 msgr2=0x7fe6cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:48.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 --2- 172.21.15.195:0/2601710129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc0f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fe6bc009960 tx=0x7fe6bc004d10).stop 2024-02-20T00:33:48.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- 172.21.15.195:0/2601710129 shutdown_connections 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 --2- 172.21.15.195:0/2601710129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc0f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- 172.21.15.195:0/2601710129 >> 172.21.15.195:0/2601710129 conn(0x7fe6cc0f1c70 msgr2=0x7fe6cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- 172.21.15.195:0/2601710129 shutdown_connections 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- 172.21.15.195:0/2601710129 wait complete. 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 Processor -- start 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- start start 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.115+0000 7fe6d4310700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc069e20 con 0x7fe6cc0f6510 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d20ac700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d20ac700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41932/0 (socket says 172.21.15.195:41932) 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d20ac700 1 -- 172.21.15.195:0/4229400155 learned_addr learned my addr 172.21.15.195:0/4229400155 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d20ac700 1 -- 172.21.15.195:0/4229400155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc009640 con 0x7fe6cc0f6510 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d20ac700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fe6bc004a10 tx=0x7fe6bc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:48.118 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc018070 con 0x7fe6cc0f6510 2024-02-20T00:33:48.119 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc069fc0 con 0x7fe6cc0f6510 2024-02-20T00:33:48.119 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6bc007e70 con 0x7fe6cc0f6510 2024-02-20T00:33:48.119 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc01d420 con 0x7fe6cc0f6510 2024-02-20T00:33:48.119 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.116+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc06a450 con 0x7fe6cc0f6510 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.117+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc059100 con 0x7fe6cc0f6510 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.117+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6bc013630 con 0x7fe6cc0f6510 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.117+0000 7fe6caffd700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 0x7fe6b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.118+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6bc0415c0 con 0x7fe6cc0f6510 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.118+0000 7fe6d18ab700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 0x7fe6b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:48.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.118+0000 7fe6d18ab700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 0x7fe6b4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe6c00096a0 tx=0x7fe6c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:48.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.122+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6bc01b9c0 con 0x7fe6cc0f6510 2024-02-20T00:33:48.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.328+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6cc0fc1f0 con 0x7fe6cc0f6510 2024-02-20T00:33:48.329 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.329+0000 7fe6caffd700 1 -- 172.21.15.195:0/4229400155 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6bc011320 con 0x7fe6cc0f6510 2024-02-20T00:33:48.329 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:48.329 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 msgr2=0x7fe6b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 0x7fe6b4032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe6c00096a0 tx=0x7fe6c0006b40).stop 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 msgr2=0x7fe6cc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fe6bc004a10 tx=0x7fe6bc004d10).stop 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 shutdown_connections 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fe6b402ff60 0x7fe6b4032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 --2- 172.21.15.195:0/4229400155 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6cc0f6510 0x7fe6cc06b9a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 >> 172.21.15.195:0/4229400155 conn(0x7fe6cc0f1c70 msgr2=0x7fe6cc0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 shutdown_connections 2024-02-20T00:33:48.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:48.330+0000 7fe6d4310700 1 -- 172.21.15.195:0/4229400155 wait complete. 2024-02-20T00:33:48.332 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:48.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:46.598917+0000 mgr.smithi087.iydsqo (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:49.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:48.328660+0000 mon.smithi087 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.195:0/4229400155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:49.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:49.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:50.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:48.599222+0000 mgr.smithi087.iydsqo (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- 172.21.15.195:0/2409283224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 msgr2=0x7f2d140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/2409283224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d140f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2d04009fb0 tx=0x7f2d04009b30).stop 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- 172.21.15.195:0/2409283224 shutdown_connections 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/2409283224 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d140f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- 172.21.15.195:0/2409283224 >> 172.21.15.195:0/2409283224 conn(0x7f2d140f0b70 msgr2=0x7f2d140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- 172.21.15.195:0/2409283224 shutdown_connections 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- 172.21.15.195:0/2409283224 wait complete. 2024-02-20T00:33:51.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 Processor -- start 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- start start 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.900+0000 7f2d1bdab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14060240 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d19b47700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d19b47700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41954/0 (socket says 172.21.15.195:41954) 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d19b47700 1 -- 172.21.15.195:0/3574688012 learned_addr learned my addr 172.21.15.195:0/3574688012 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d19b47700 1 -- 172.21.15.195:0/3574688012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d04005040 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d19b47700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f2d04005010 tx=0x7f2d0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d04009320 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14060440 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d04013380 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d04014e50 con 0x7f2d140f5410 2024-02-20T00:33:51.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.901+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d14109b90 con 0x7f2d140f5410 2024-02-20T00:33:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.902+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d040263a0 con 0x7f2d140f5410 2024-02-20T00:33:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.902+0000 7f2d0a7fc700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 0x7f2d000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.902+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d04040fc0 con 0x7f2d140f5410 2024-02-20T00:33:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.902+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d14045ad0 con 0x7f2d140f5410 2024-02-20T00:33:51.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.902+0000 7f2d19346700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 0x7f2d000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:51.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.903+0000 7f2d19346700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 0x7f2d000323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2d100096a0 tx=0x7f2d10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:51.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:51.907+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d04012920 con 0x7f2d140f5410 2024-02-20T00:33:52.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.112+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d1410a3b0 con 0x7f2d140f5410 2024-02-20T00:33:52.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.113+0000 7f2d0a7fc700 1 -- 172.21.15.195:0/3574688012 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d0402b020 con 0x7f2d140f5410 2024-02-20T00:33:52.114 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:52.115 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:52.116 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 msgr2=0x7f2d000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 0x7f2d000323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f2d100096a0 tx=0x7f2d10006b40).stop 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 msgr2=0x7f2d141094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f2d04005010 tx=0x7f2d0400b040).stop 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 shutdown_connections 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2d0002ff10 0x7f2d000323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 --2- 172.21.15.195:0/3574688012 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d140f5410 0x7f2d141094e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 >> 172.21.15.195:0/3574688012 conn(0x7f2d140f0b70 msgr2=0x7f2d140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 shutdown_connections 2024-02-20T00:33:52.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:52.115+0000 7f2d1bdab700 1 -- 172.21.15.195:0/3574688012 wait complete. 2024-02-20T00:33:52.118 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:52.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:50.599463+0000 mgr.smithi087.iydsqo (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:52.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:52.114168+0000 mon.smithi087 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.195:0/3574688012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:53.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:53.771 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:52.599826+0000 mgr.smithi087.iydsqo (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 -- 172.21.15.195:0/4284211512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 msgr2=0x7f3bbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 --2- 172.21.15.195:0/4284211512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc0f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f3bb0009fb0 tx=0x7f3bb0009b30).stop 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 -- 172.21.15.195:0/4284211512 shutdown_connections 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 --2- 172.21.15.195:0/4284211512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc0f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 -- 172.21.15.195:0/4284211512 >> 172.21.15.195:0/4284211512 conn(0x7f3bbc0f1c70 msgr2=0x7f3bbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 -- 172.21.15.195:0/4284211512 shutdown_connections 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.655+0000 7f3bc092e700 1 -- 172.21.15.195:0/4284211512 wait complete. 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bc092e700 1 Processor -- start 2024-02-20T00:33:55.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bc092e700 1 -- start start 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bc092e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bc092e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc106cd0 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bba59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bba59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41972/0 (socket says 172.21.15.195:41972) 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.656+0000 7f3bba59c700 1 -- 172.21.15.195:0/3360322109 learned_addr learned my addr 172.21.15.195:0/3360322109 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3bba59c700 1 -- 172.21.15.195:0/3360322109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb0005040 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3bba59c700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f3bb0005010 tx=0x7f3bb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb0009320 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bb0013380 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc106ed0 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bb0014e50 con 0x7f3bbc0f6510 2024-02-20T00:33:55.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.657+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc107360 con 0x7f3bbc0f6510 2024-02-20T00:33:55.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bb00095f0 con 0x7f3bbc0f6510 2024-02-20T00:33:55.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc045ad0 con 0x7f3bbc0f6510 2024-02-20T00:33:55.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3baaffd700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:55.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bb0040f10 con 0x7f3bbc0f6510 2024-02-20T00:33:55.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3bb9d9b700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:55.662 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.658+0000 7f3bb9d9b700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3bac0096a0 tx=0x7f3bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:55.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.663+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bb0014420 con 0x7f3bbc0f6510 2024-02-20T00:33:55.867 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.866+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bbc108210 con 0x7f3bbc0f6510 2024-02-20T00:33:55.868 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.866+0000 7f3baaffd700 1 -- 172.21.15.195:0/3360322109 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bb0010230 con 0x7f3bbc0f6510 2024-02-20T00:33:55.868 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:55.868 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 msgr2=0x7f3ba40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3bac0096a0 tx=0x7f3bac006b40).stop 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 msgr2=0x7f3bbc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f3bb0005010 tx=0x7f3bb000b040).stop 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 shutdown_connections 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 --2- 172.21.15.195:0/3360322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 >> 172.21.15.195:0/3360322109 conn(0x7f3bbc0f1c70 msgr2=0x7f3bbc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 shutdown_connections 2024-02-20T00:33:55.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:55.868+0000 7f3bc092e700 1 -- 172.21.15.195:0/3360322109 wait complete. 2024-02-20T00:33:55.871 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:33:56.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:54.600186+0000 mgr.smithi087.iydsqo (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:56.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:55.867743+0000 mon.smithi087 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.195:0/3360322109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:33:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:56.672112+0000 mon.smithi087 (mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:33:57.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:56.672432+0000 mon.smithi087 (mon.0) 261 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:33:57.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:33:57.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:33:58.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:33:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:56.600549+0000 mgr.smithi087.iydsqo (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:33:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- 172.21.15.195:0/2008363051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 msgr2=0x7ff6d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 --2- 172.21.15.195:0/2008363051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d40f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff6d0009fb0 tx=0x7ff6d0009b30).stop 2024-02-20T00:33:59.657 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- 172.21.15.195:0/2008363051 shutdown_connections 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 --2- 172.21.15.195:0/2008363051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d40f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- 172.21.15.195:0/2008363051 >> 172.21.15.195:0/2008363051 conn(0x7ff6d40f0b70 msgr2=0x7ff6d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- 172.21.15.195:0/2008363051 shutdown_connections 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- 172.21.15.195:0/2008363051 wait complete. 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 Processor -- start 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 -- start start 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.654+0000 7ff6db6da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6db6da700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4106cc0 con 0x7ff6d40f5410 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6d9476700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6d9476700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59268/0 (socket says 172.21.15.195:59268) 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6d9476700 1 -- 172.21.15.195:0/936643565 learned_addr learned my addr 172.21.15.195:0/936643565 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6d9476700 1 -- 172.21.15.195:0/936643565 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0005040 con 0x7ff6d40f5410 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6d9476700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff6d0005010 tx=0x7ff6d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:59.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d0009320 con 0x7ff6d40f5410 2024-02-20T00:33:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6d0013380 con 0x7ff6d40f5410 2024-02-20T00:33:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d0014e50 con 0x7ff6d40f5410 2024-02-20T00:33:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.655+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d4106ec0 con 0x7ff6d40f5410 2024-02-20T00:33:59.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.656+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d4107350 con 0x7ff6d40f5410 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.656+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6d00095f0 con 0x7ff6d40f5410 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.657+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d4045ad0 con 0x7ff6d40f5410 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.657+0000 7ff6c9ffb700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.657+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6d0040ee0 con 0x7ff6d40f5410 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.657+0000 7ff6d8c75700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:33:59.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.657+0000 7ff6d8c75700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff6c40096a0 tx=0x7ff6c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:33:59.663 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.662+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6d0014420 con 0x7ff6d40f5410 2024-02-20T00:33:59.868 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.866+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6d4108200 con 0x7ff6d40f5410 2024-02-20T00:33:59.868 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.867+0000 7ff6c9ffb700 1 -- 172.21.15.195:0/936643565 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6d001d980 con 0x7ff6d40f5410 2024-02-20T00:33:59.868 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:33:59.869 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:33:59.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.868+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 msgr2=0x7ff6c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:59.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.868+0000 7ff6db6da700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff6c40096a0 tx=0x7ff6c4006b40).stop 2024-02-20T00:33:59.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.868+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 msgr2=0x7ff6d41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:33:59.870 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.868+0000 7ff6db6da700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff6d0005010 tx=0x7ff6d000b040).stop 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 shutdown_connections 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 --2- 172.21.15.195:0/936643565 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff6d40f5410 0x7ff6d41067d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 >> 172.21.15.195:0/936643565 conn(0x7ff6d40f0b70 msgr2=0x7ff6d405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 shutdown_connections 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:33:59.869+0000 7ff6db6da700 1 -- 172.21.15.195:0/936643565 wait complete. 2024-02-20T00:33:59.871 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:00.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:33:58.600865+0000 mgr.smithi087.iydsqo (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:00.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:33:59.868037+0000 mon.smithi087 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.195:0/936643565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:01.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:01.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:02.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:00.601206+0000 mgr.smithi087.iydsqo (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:03.382 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 -- 172.21.15.195:0/3265051240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 msgr2=0x7fa1240f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:03.382 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 --2- 172.21.15.195:0/3265051240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1240f4fa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa114009fb0 tx=0x7fa114009b30).stop 2024-02-20T00:34:03.382 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 -- 172.21.15.195:0/3265051240 shutdown_connections 2024-02-20T00:34:03.382 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 --2- 172.21.15.195:0/3265051240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1240f4fa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:03.382 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 -- 172.21.15.195:0/3265051240 >> 172.21.15.195:0/3265051240 conn(0x7fa1240f02e0 msgr2=0x7fa1240f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 -- 172.21.15.195:0/3265051240 shutdown_connections 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.379+0000 7fa12b328700 1 -- 172.21.15.195:0/3265051240 wait complete. 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa12b328700 1 Processor -- start 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa12b328700 1 -- start start 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa12b328700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa12b328700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa124106cb0 con 0x7fa1240f4b80 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa1290c4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa1290c4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:59284/0 (socket says 172.21.15.195:59284) 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa1290c4700 1 -- 172.21.15.195:0/4019733381 learned_addr learned my addr 172.21.15.195:0/4019733381 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa1290c4700 1 -- 172.21.15.195:0/4019733381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114005040 con 0x7fa1240f4b80 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.380+0000 7fa1290c4700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa114005010 tx=0x7fa11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114009320 con 0x7fa1240f4b80 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa124106eb0 con 0x7fa1240f4b80 2024-02-20T00:34:03.383 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa114013380 con 0x7fa1240f4b80 2024-02-20T00:34:03.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114014e50 con 0x7fa1240f4b80 2024-02-20T00:34:03.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa124107340 con 0x7fa1240f4b80 2024-02-20T00:34:03.384 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.381+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa114014420 con 0x7fa1240f4b80 2024-02-20T00:34:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.382+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa124045ad0 con 0x7fa1240f4b80 2024-02-20T00:34:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.382+0000 7fa119ffb700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.382+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa114040e60 con 0x7fa1240f4b80 2024-02-20T00:34:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.382+0000 7fa1288c3700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:03.385 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.382+0000 7fa1288c3700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 0x7fa1100323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:03.388 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.386+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1140095f0 con 0x7fa1240f4b80 2024-02-20T00:34:03.589 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.588+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa124108960 con 0x7fa1240f4b80 2024-02-20T00:34:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.589+0000 7fa119ffb700 1 -- 172.21.15.195:0/4019733381 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa11401db70 con 0x7fa1240f4b80 2024-02-20T00:34:03.590 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:03.590 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 msgr2=0x7fa1100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 0x7fa1100323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).stop 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 msgr2=0x7fa1241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa114005010 tx=0x7fa11400b040).stop 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 shutdown_connections 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 --2- 172.21.15.195:0/4019733381 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1240f4b80 0x7fa1241067c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 >> 172.21.15.195:0/4019733381 conn(0x7fa1240f02e0 msgr2=0x7fa1240f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 shutdown_connections 2024-02-20T00:34:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:03.590+0000 7fa12b328700 1 -- 172.21.15.195:0/4019733381 wait complete. 2024-02-20T00:34:03.593 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:04.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:02.601600+0000 mgr.smithi087.iydsqo (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:03.589984+0000 mon.smithi087 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.195:0/4019733381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:05.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:05.299 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:06.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:04.601925+0000 mgr.smithi087.iydsqo (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 -- 172.21.15.195:0/90351186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 msgr2=0x7f27680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 --2- 172.21.15.195:0/90351186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27680f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2758009fb0 tx=0x7f2758009b30).stop 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 -- 172.21.15.195:0/90351186 shutdown_connections 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 --2- 172.21.15.195:0/90351186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27680f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 -- 172.21.15.195:0/90351186 >> 172.21.15.195:0/90351186 conn(0x7f27680f09a0 msgr2=0x7f27680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:06.991 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 -- 172.21.15.195:0/90351186 shutdown_connections 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.988+0000 7f276e39d700 1 -- 172.21.15.195:0/90351186 wait complete. 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276e39d700 1 Processor -- start 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276e39d700 1 -- start start 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276e39d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276e39d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27681059c0 con 0x7f27680f51a0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276d39b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276d39b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60224/0 (socket says 172.21.15.195:60224) 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276d39b700 1 -- 172.21.15.195:0/398066569 learned_addr learned my addr 172.21.15.195:0/398066569 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276d39b700 1 -- 172.21.15.195:0/398066569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2758005040 con 0x7f27680f51a0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.989+0000 7f276d39b700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f27580089f0 tx=0x7f275800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.990+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2758009320 con 0x7f27680f51a0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.990+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2768105bc0 con 0x7f27680f51a0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.990+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2758013380 con 0x7f27680f51a0 2024-02-20T00:34:06.992 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.990+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2758014e20 con 0x7f27680f51a0 2024-02-20T00:34:06.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.990+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2768106050 con 0x7f27680f51a0 2024-02-20T00:34:06.993 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2768045ad0 con 0x7f27680f51a0 2024-02-20T00:34:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27580095f0 con 0x7f27680f51a0 2024-02-20T00:34:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f275dffb700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 0x7f27540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2758040f10 con 0x7f27680f51a0 2024-02-20T00:34:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f276cb9a700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 0x7f27540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:06.994 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.991+0000 7f276cb9a700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 0x7f27540323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f27640096a0 tx=0x7f2764006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:06.997 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:06.995+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2758014420 con 0x7f27680f51a0 2024-02-20T00:34:07.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.201+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2768106f00 con 0x7f27680f51a0 2024-02-20T00:34:07.203 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.202+0000 7f275dffb700 1 -- 172.21.15.195:0/398066569 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f275801d980 con 0x7f27680f51a0 2024-02-20T00:34:07.203 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:07.204 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 msgr2=0x7f27540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 0x7f27540323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f27640096a0 tx=0x7f2764006b40).stop 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 msgr2=0x7f27681054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f27580089f0 tx=0x7f275800b040).stop 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 shutdown_connections 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f275402ff10 0x7f27540323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 --2- 172.21.15.195:0/398066569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f27680f51a0 0x7f27681054d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:07.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 >> 172.21.15.195:0/398066569 conn(0x7f27680f09a0 msgr2=0x7f27680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 shutdown_connections 2024-02-20T00:34:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:07.203+0000 7f276e39d700 1 -- 172.21.15.195:0/398066569 wait complete. 2024-02-20T00:34:07.206 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:07.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:07 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:07.203055+0000 mon.smithi087 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.195:0/398066569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:08.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:06.602204+0000 mgr.smithi087.iydsqo (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:08.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:08.792 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:10.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:08.602551+0000 mgr.smithi087.iydsqo (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:10.552 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 -- 172.21.15.195:0/3936371963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 msgr2=0x7f06240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 --2- 172.21.15.195:0/3936371963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0614009fb0 tx=0x7f0614009b30).stop 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 -- 172.21.15.195:0/3936371963 shutdown_connections 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 --2- 172.21.15.195:0/3936371963 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 -- 172.21.15.195:0/3936371963 >> 172.21.15.195:0/3936371963 conn(0x7f06240f0b70 msgr2=0x7f06240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 -- 172.21.15.195:0/3936371963 shutdown_connections 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.549+0000 7f062c568700 1 -- 172.21.15.195:0/3936371963 wait complete. 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062c568700 1 Processor -- start 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062c568700 1 -- start start 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062c568700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062c568700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06240fb960 con 0x7f06240f5410 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062a304700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062a304700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60248/0 (socket says 172.21.15.195:60248) 2024-02-20T00:34:10.553 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062a304700 1 -- 172.21.15.195:0/4017726667 learned_addr learned my addr 172.21.15.195:0/4017726667 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.550+0000 7f062a304700 1 -- 172.21.15.195:0/4017726667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614005040 con 0x7f06240f5410 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f062a304700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0614005010 tx=0x7f061400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0614009320 con 0x7f06240f5410 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06240fbb00 con 0x7f06240f5410 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0614013380 con 0x7f06240f5410 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0614014e50 con 0x7f06240f5410 2024-02-20T00:34:10.554 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.551+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06240fbf90 con 0x7f06240f5410 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06140095f0 con 0x7f06240f5410 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0624059100 con 0x7f06240f5410 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f061affd700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 0x7f06100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0614040fa0 con 0x7f06240f5410 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f0629b03700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 0x7f06100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:10.555 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.552+0000 7f0629b03700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 0x7f06100323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f06200096a0 tx=0x7f0620006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:10.558 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.557+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0614014420 con 0x7f06240f5410 2024-02-20T00:34:10.762 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.761+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06240fafa0 con 0x7f06240f5410 2024-02-20T00:34:10.763 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.761+0000 7f061affd700 1 -- 172.21.15.195:0/4017726667 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0614010230 con 0x7f06240f5410 2024-02-20T00:34:10.763 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:10.763 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 msgr2=0x7f06100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 0x7f06100323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f06200096a0 tx=0x7f0620006b40).stop 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 msgr2=0x7f06240fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0614005010 tx=0x7f061400b040).stop 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 shutdown_connections 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061002ff10 0x7f06100323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 --2- 172.21.15.195:0/4017726667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06240f5410 0x7f06240fd4e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:10.765 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 >> 172.21.15.195:0/4017726667 conn(0x7f06240f0b70 msgr2=0x7f062405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 shutdown_connections 2024-02-20T00:34:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:10.763+0000 7f062c568700 1 -- 172.21.15.195:0/4017726667 wait complete. 2024-02-20T00:34:10.766 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:11.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:10.762625+0000 mon.smithi087 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.195:0/4017726667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:12.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:12.419 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:12.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:10.602925+0000 mgr.smithi087.iydsqo (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- 172.21.15.195:0/1150353932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd340f6a00 msgr2=0x7efd340f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/1150353932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd340f6a00 0x7efd340f6e20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7efd24009fb0 tx=0x7efd24009b30).stop 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- 172.21.15.195:0/1150353932 shutdown_connections 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/1150353932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd340f6a00 0x7efd340f6e20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- 172.21.15.195:0/1150353932 >> 172.21.15.195:0/1150353932 conn(0x7efd340f0260 msgr2=0x7efd340f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:14.330 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- 172.21.15.195:0/1150353932 shutdown_connections 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- 172.21.15.195:0/1150353932 wait complete. 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 Processor -- start 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.313+0000 7efd3b5e8700 1 -- start start 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd3b5e8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd3b5e8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd34106090 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd39384700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd39384700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60274/0 (socket says 172.21.15.195:60274) 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd39384700 1 -- 172.21.15.195:0/4114520056 learned_addr learned my addr 172.21.15.195:0/4114520056 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd39384700 1 -- 172.21.15.195:0/4114520056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24005040 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd39384700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7efd24009f80 tx=0x7efd24009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd24013380 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd34106290 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.314+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd24013790 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.315+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd24014420 con 0x7efd34105780 2024-02-20T00:34:14.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.315+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd34106de0 con 0x7efd34105780 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.315+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd24025060 con 0x7efd34105780 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.315+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd34059100 con 0x7efd34105780 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.316+0000 7efd29ffb700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 0x7efd20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.316+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd24040e00 con 0x7efd34105780 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.316+0000 7efd38b83700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 0x7efd20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.316+0000 7efd38b83700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 0x7efd20032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efd300096a0 tx=0x7efd30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:14.332 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.320+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd2400f280 con 0x7efd34105780 2024-02-20T00:34:14.530 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.528+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd341075a0 con 0x7efd34105780 2024-02-20T00:34:14.531 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.529+0000 7efd29ffb700 1 -- 172.21.15.195:0/4114520056 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd2400f280 con 0x7efd34105780 2024-02-20T00:34:14.531 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:14.531 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:12.603280+0000 mgr.smithi087.iydsqo (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 msgr2=0x7efd20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 0x7efd20032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efd300096a0 tx=0x7efd30006b40).stop 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 msgr2=0x7efd34105ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7efd24009f80 tx=0x7efd24009b30).stop 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 shutdown_connections 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7efd2002ff60 0x7efd20032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 --2- 172.21.15.195:0/4114520056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efd34105780 0x7efd34105ba0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 >> 172.21.15.195:0/4114520056 conn(0x7efd340f0260 msgr2=0x7efd340f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 shutdown_connections 2024-02-20T00:34:14.533 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:14.531+0000 7efd3b5e8700 1 -- 172.21.15.195:0/4114520056 wait complete. 2024-02-20T00:34:14.534 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:15.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:15 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:14.530187+0000 mon.smithi087 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.195:0/4114520056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:16.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:16.093 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:14.603614+0000 mgr.smithi087.iydsqo (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:17.998 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 -- 172.21.15.195:0/3577056067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 msgr2=0x7fef500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 --2- 172.21.15.195:0/3577056067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef500f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fef40009fb0 tx=0x7fef40009b30).stop 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 -- 172.21.15.195:0/3577056067 shutdown_connections 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 --2- 172.21.15.195:0/3577056067 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef500f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 -- 172.21.15.195:0/3577056067 >> 172.21.15.195:0/3577056067 conn(0x7fef500f0b70 msgr2=0x7fef500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 -- 172.21.15.195:0/3577056067 shutdown_connections 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.995+0000 7fef5860d700 1 -- 172.21.15.195:0/3577056067 wait complete. 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef5860d700 1 Processor -- start 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef5860d700 1 -- start start 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef5860d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef5860d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef50060280 con 0x7fef500f5410 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef563a9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:17.999 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef563a9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40764/0 (socket says 172.21.15.195:40764) 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.996+0000 7fef563a9700 1 -- 172.21.15.195:0/3754697773 learned_addr learned my addr 172.21.15.195:0/3754697773 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef563a9700 1 -- 172.21.15.195:0/3754697773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef40005040 con 0x7fef500f5410 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef563a9700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fef40005010 tx=0x7fef4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40009320 con 0x7fef500f5410 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef40013380 con 0x7fef500f5410 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef50060480 con 0x7fef500f5410 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40014e50 con 0x7fef500f5410 2024-02-20T00:34:18.000 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.997+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef50109bd0 con 0x7fef500f5410 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef400095f0 con 0x7fef500f5410 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef50045ad0 con 0x7fef500f5410 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef46ffd700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 0x7fef3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef40040fa0 con 0x7fef500f5410 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef55ba8700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 0x7fef3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:18.001 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:17.998+0000 7fef55ba8700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 0x7fef3c0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fef4c0096a0 tx=0x7fef4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:18.004 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.002+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef40014420 con 0x7fef500f5410 2024-02-20T00:34:18.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.208+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef500fafa0 con 0x7fef500f5410 2024-02-20T00:34:18.210 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.209+0000 7fef46ffd700 1 -- 172.21.15.195:0/3754697773 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef40010230 con 0x7fef500f5410 2024-02-20T00:34:18.210 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:18.210 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 msgr2=0x7fef3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 0x7fef3c0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fef4c0096a0 tx=0x7fef4c006b40).stop 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 msgr2=0x7fef501094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fef40005010 tx=0x7fef4000b040).stop 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 shutdown_connections 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fef3c02ff10 0x7fef3c0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:18.212 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 --2- 172.21.15.195:0/3754697773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fef500f5410 0x7fef501094a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 >> 172.21.15.195:0/3754697773 conn(0x7fef500f0b70 msgr2=0x7fef500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 shutdown_connections 2024-02-20T00:34:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:18.210+0000 7fef5860d700 1 -- 172.21.15.195:0/3754697773 wait complete. 2024-02-20T00:34:18.213 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:18.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:16.603910+0000 mgr.smithi087.iydsqo (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:18.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:18.209977+0000 mon.smithi087 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.195:0/3754697773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:19.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:19.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:20.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:18.604260+0000 mgr.smithi087.iydsqo (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:21.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 -- 172.21.15.195:0/1447674828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 msgr2=0x7ff1640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/1447674828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff154009fb0 tx=0x7ff154009b30).stop 2024-02-20T00:34:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 -- 172.21.15.195:0/1447674828 shutdown_connections 2024-02-20T00:34:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/1447674828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 -- 172.21.15.195:0/1447674828 >> 172.21.15.195:0/1447674828 conn(0x7ff1640f0b70 msgr2=0x7ff1640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:21.744 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 -- 172.21.15.195:0/1447674828 shutdown_connections 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.739+0000 7ff16b8a7700 1 -- 172.21.15.195:0/1447674828 wait complete. 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff16b8a7700 1 Processor -- start 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff16b8a7700 1 -- start start 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff16b8a7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff16b8a7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1640fb960 con 0x7ff1640f5410 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff169643700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff169643700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40784/0 (socket says 172.21.15.195:40784) 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff169643700 1 -- 172.21.15.195:0/2207629484 learned_addr learned my addr 172.21.15.195:0/2207629484 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff169643700 1 -- 172.21.15.195:0/2207629484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154005040 con 0x7ff1640f5410 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.740+0000 7ff169643700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff154005010 tx=0x7ff15400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.741+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff154009320 con 0x7ff1640f5410 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.741+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1640fbb00 con 0x7ff1640f5410 2024-02-20T00:34:21.745 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.741+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff154013380 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.741+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff154014e50 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.741+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1640fbf90 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1540263a0 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff159ffb700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 0x7ff1500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff154040fa0 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff164059100 con 0x7ff1640f5410 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff168e42700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 0x7ff1500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:21.746 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.742+0000 7ff168e42700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 0x7ff1500323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff1600096a0 tx=0x7ff160006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:21.748 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.747+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff154012920 con 0x7ff1640f5410 2024-02-20T00:34:21.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.951+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1640fc690 con 0x7ff1640f5410 2024-02-20T00:34:21.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.952+0000 7ff159ffb700 1 -- 172.21.15.195:0/2207629484 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff15402b020 con 0x7ff1640f5410 2024-02-20T00:34:21.954 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:21.954 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 msgr2=0x7ff1500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 0x7ff1500323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff1600096a0 tx=0x7ff160006b40).stop 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 msgr2=0x7ff1640fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff154005010 tx=0x7ff15400b040).stop 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 shutdown_connections 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff15002ff10 0x7ff1500323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 --2- 172.21.15.195:0/2207629484 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1640f5410 0x7ff1640fd4e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 >> 172.21.15.195:0/2207629484 conn(0x7ff1640f0b70 msgr2=0x7ff16405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 shutdown_connections 2024-02-20T00:34:21.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:21.954+0000 7ff16b8a7700 1 -- 172.21.15.195:0/2207629484 wait complete. 2024-02-20T00:34:21.957 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:22.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:20.604610+0000 mgr.smithi087.iydsqo (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:22.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:21.953517+0000 mon.smithi087 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.195:0/2207629484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:23.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:23.560 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:24.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:22.605008+0000 mgr.smithi087.iydsqo (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 -- 172.21.15.195:0/4030358972 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 msgr2=0x7f29180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 --2- 172.21.15.195:0/4030358972 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f29180f66c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f2914009fb0 tx=0x7f2914009b30).stop 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 -- 172.21.15.195:0/4030358972 shutdown_connections 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 --2- 172.21.15.195:0/4030358972 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f29180f66c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 -- 172.21.15.195:0/4030358972 >> 172.21.15.195:0/4030358972 conn(0x7f29180f1aa0 msgr2=0x7f29180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 -- 172.21.15.195:0/4030358972 shutdown_connections 2024-02-20T00:34:25.299 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 -- 172.21.15.195:0/4030358972 wait complete. 2024-02-20T00:34:25.301 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.297+0000 7f291f1c2700 1 Processor -- start 2024-02-20T00:34:25.301 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291f1c2700 1 -- start start 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291f1c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291f1c2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2918069e20 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291e1c0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291e1c0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40802/0 (socket says 172.21.15.195:40802) 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291e1c0700 1 -- 172.21.15.195:0/3290887770 learned_addr learned my addr 172.21.15.195:0/3290887770 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291e1c0700 1 -- 172.21.15.195:0/3290887770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2914005040 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.298+0000 7f291e1c0700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f29140089f0 tx=0x7f291400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2914009320 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2914013380 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2914014e20 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2918069fc0 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291806a450 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.299+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29140095f0 con 0x7f29180f62a0 2024-02-20T00:34:25.302 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.300+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2918059100 con 0x7f29180f62a0 2024-02-20T00:34:25.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.300+0000 7f290effd700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:25.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.300+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2914040ec0 con 0x7f29180f62a0 2024-02-20T00:34:25.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.300+0000 7f291d9bf700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:25.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.300+0000 7f291d9bf700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 0x7f2904032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f29080096a0 tx=0x7f2908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:25.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.304+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2914014420 con 0x7f29180f62a0 2024-02-20T00:34:25.516 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.514+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29180fbe30 con 0x7f29180f62a0 2024-02-20T00:34:25.516 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.514+0000 7f290effd700 1 -- 172.21.15.195:0/3290887770 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2914010230 con 0x7f29180f62a0 2024-02-20T00:34:25.517 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:25.517 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:25.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 msgr2=0x7f2904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:25.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 0x7f2904032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f29080096a0 tx=0x7f2908006b40).stop 2024-02-20T00:34:25.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 msgr2=0x7f291806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:25.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f29140089f0 tx=0x7f291400b040).stop 2024-02-20T00:34:25.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 shutdown_connections 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f290402ff60 0x7f2904032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 --2- 172.21.15.195:0/3290887770 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29180f62a0 0x7f291806b9a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 >> 172.21.15.195:0/3290887770 conn(0x7f29180f1aa0 msgr2=0x7f29180f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 shutdown_connections 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:25.516+0000 7f291f1c2700 1 -- 172.21.15.195:0/3290887770 wait complete. 2024-02-20T00:34:25.519 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:24.605313+0000 mgr.smithi087.iydsqo (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:25.515639+0000 mon.smithi087 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.195:0/3290887770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:27.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:27.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:28.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:26.605576+0000 mgr.smithi087.iydsqo (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:29.112 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.069+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/807996532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 msgr2=0x7f7bc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.069+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/807996532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc80f66c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f7bbc009fb0 tx=0x7f7bbc009b30).stop 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.069+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/807996532 shutdown_connections 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.069+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/807996532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc80f66c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.069+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/807996532 >> 172.21.15.195:0/807996532 conn(0x7f7bc80f1aa0 msgr2=0x7f7bc80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/807996532 shutdown_connections 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/807996532 wait complete. 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 Processor -- start 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 -- start start 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:29.113 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.070+0000 7f7bcc8bc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc8069e20 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bc6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bc6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:48430/0 (socket says 172.21.15.195:48430) 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bc6d9d700 1 -- 172.21.15.195:0/297907864 learned_addr learned my addr 172.21.15.195:0/297907864 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bc6d9d700 1 -- 172.21.15.195:0/297907864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc005040 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bc6d9d700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7bbc0089f0 tx=0x7f7bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc009320 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc8069fc0 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bbc013380 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bbc014e20 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.071+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc806a450 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.072+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc8059100 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.072+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bbc0095f0 con 0x7f7bc80f62a0 2024-02-20T00:34:29.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.072+0000 7f7baf7fe700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 0x7f7bb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:29.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.072+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bbc040e90 con 0x7f7bc80f62a0 2024-02-20T00:34:29.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.072+0000 7f7bc659c700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 0x7f7bb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:29.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.073+0000 7f7bc659c700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 0x7f7bb00323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7bb80096a0 tx=0x7f7bb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:29.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.077+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bbc014420 con 0x7f7bc80f62a0 2024-02-20T00:34:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.281+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bc80fbe30 con 0x7f7bc80f62a0 2024-02-20T00:34:29.283 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.282+0000 7f7baf7fe700 1 -- 172.21.15.195:0/297907864 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bbc01d980 con 0x7f7bc80f62a0 2024-02-20T00:34:29.284 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:29.284 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 msgr2=0x7f7bb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 0x7f7bb00323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7bb80096a0 tx=0x7f7bb8006b40).stop 2024-02-20T00:34:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 msgr2=0x7f7bc806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7bbc0089f0 tx=0x7f7bbc00b040).stop 2024-02-20T00:34:29.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 shutdown_connections 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7bb002ff10 0x7f7bb00323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 --2- 172.21.15.195:0/297907864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bc80f62a0 0x7f7bc806b9a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 >> 172.21.15.195:0/297907864 conn(0x7f7bc80f1aa0 msgr2=0x7f7bc80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 shutdown_connections 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:29.283+0000 7f7bcc8bc700 1 -- 172.21.15.195:0/297907864 wait complete. 2024-02-20T00:34:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:30.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:28.605913+0000 mgr.smithi087.iydsqo (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:30.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:29.283055+0000 mon.smithi087 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.195:0/297907864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:30.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:30.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:32.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:30.606258+0000 mgr.smithi087.iydsqo (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/2005987718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 msgr2=0x7f9a180fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/2005987718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a180fb4f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f9a08009fb0 tx=0x7f9a08009b30).stop 2024-02-20T00:34:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/2005987718 shutdown_connections 2024-02-20T00:34:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/2005987718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a180fb4f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:32.735 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/2005987718 >> 172.21.15.195:0/2005987718 conn(0x7f9a180f0b70 msgr2=0x7f9a180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/2005987718 shutdown_connections 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/2005987718 wait complete. 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 Processor -- start 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.732+0000 7f9a1e9e6700 1 -- start start 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a1e9e6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a1e9e6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a18108e70 con 0x7f9a18060280 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a17fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a17fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:48446/0 (socket says 172.21.15.195:48446) 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a17fff700 1 -- 172.21.15.195:0/3579878672 learned_addr learned my addr 172.21.15.195:0/3579878672 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a17fff700 1 -- 172.21.15.195:0/3579878672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a08005040 con 0x7f9a18060280 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a17fff700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9a080089f0 tx=0x7f9a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a08009320 con 0x7f9a18060280 2024-02-20T00:34:32.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a18109070 con 0x7f9a18060280 2024-02-20T00:34:32.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a08013380 con 0x7f9a18060280 2024-02-20T00:34:32.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a08014e20 con 0x7f9a18060280 2024-02-20T00:34:32.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.733+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a18109500 con 0x7f9a18060280 2024-02-20T00:34:32.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.734+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a08014420 con 0x7f9a18060280 2024-02-20T00:34:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.734+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a18045ad0 con 0x7f9a18060280 2024-02-20T00:34:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.734+0000 7f9a157fa700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 0x7f9a040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.734+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a08040ff0 con 0x7f9a18060280 2024-02-20T00:34:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.735+0000 7f9a0ffff700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 0x7f9a040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:32.738 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.735+0000 7f9a0ffff700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 0x7f9a040323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9a000096a0 tx=0x7f9a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:32.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.739+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a080095f0 con 0x7f9a18060280 2024-02-20T00:34:32.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.943+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a1810a3b0 con 0x7f9a18060280 2024-02-20T00:34:32.945 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.944+0000 7f9a157fa700 1 -- 172.21.15.195:0/3579878672 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a0801db70 con 0x7f9a18060280 2024-02-20T00:34:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:32.946 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:32.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 msgr2=0x7f9a040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:32.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 0x7f9a040323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9a000096a0 tx=0x7f9a00006b40).stop 2024-02-20T00:34:32.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 msgr2=0x7f9a18108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:32.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9a080089f0 tx=0x7f9a0800b040).stop 2024-02-20T00:34:32.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 shutdown_connections 2024-02-20T00:34:32.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f9a0402ff10 0x7f9a040323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:32.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 --2- 172.21.15.195:0/3579878672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9a18060280 0x7f9a18108980 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:32.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 >> 172.21.15.195:0/3579878672 conn(0x7f9a180f0b70 msgr2=0x7f9a180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:32.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.945+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 shutdown_connections 2024-02-20T00:34:32.948 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:32.946+0000 7f9a1e9e6700 1 -- 172.21.15.195:0/3579878672 wait complete. 2024-02-20T00:34:32.951 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:33.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:32.945293+0000 mon.smithi087 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.195:0/3579878672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:32.606593+0000 mgr.smithi087.iydsqo (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:34.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:34.712 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:36.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:34.606998+0000 mgr.smithi087.iydsqo (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:36.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 -- 172.21.15.195:0/3253034870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 msgr2=0x7fa2fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:36.683 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3253034870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc0f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fa2ec009fb0 tx=0x7fa2ec009b30).stop 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 -- 172.21.15.195:0/3253034870 shutdown_connections 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3253034870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc0f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 -- 172.21.15.195:0/3253034870 >> 172.21.15.195:0/3253034870 conn(0x7fa2fc0f0b70 msgr2=0x7fa2fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 -- 172.21.15.195:0/3253034870 shutdown_connections 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.681+0000 7fa300aa7700 1 -- 172.21.15.195:0/3253034870 wait complete. 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.682+0000 7fa300aa7700 1 Processor -- start 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.682+0000 7fa300aa7700 1 -- start start 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.682+0000 7fa300aa7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa300aa7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc105c10 con 0x7fa2fc0f5410 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53954/0 (socket says 172.21.15.195:53954) 2024-02-20T00:34:36.684 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2fa59c700 1 -- 172.21.15.195:0/3846598623 learned_addr learned my addr 172.21.15.195:0/3846598623 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2fa59c700 1 -- 172.21.15.195:0/3846598623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec005040 con 0x7fa2fc0f5410 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2fa59c700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa2ec005010 tx=0x7fa2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.683+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec009320 con 0x7fa2fc0f5410 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc105e10 con 0x7fa2fc0f5410 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2ec013380 con 0x7fa2fc0f5410 2024-02-20T00:34:36.686 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec014e50 con 0x7fa2fc0f5410 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc1062a0 con 0x7fa2fc0f5410 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2ec0095f0 con 0x7fa2fc0f5410 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc045ad0 con 0x7fa2fc0f5410 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.684+0000 7fa2f2ffd700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.685+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2ec040f10 con 0x7fa2fc0f5410 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.685+0000 7fa2f9d9b700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:36.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.685+0000 7fa2f9d9b700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:36.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.689+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2ec014420 con 0x7fa2fc0f5410 2024-02-20T00:34:36.892 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.891+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2fc107150 con 0x7fa2fc0f5410 2024-02-20T00:34:36.893 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.892+0000 7fa2f2ffd700 1 -- 172.21.15.195:0/3846598623 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2ec010230 con 0x7fa2fc0f5410 2024-02-20T00:34:36.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:36.893 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 msgr2=0x7fa2e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa2e40096a0 tx=0x7fa2e4006b40).stop 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 msgr2=0x7fa2fc105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa2ec005010 tx=0x7fa2ec00b040).stop 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 shutdown_connections 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 --2- 172.21.15.195:0/3846598623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2fc0f5410 0x7fa2fc105720 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 >> 172.21.15.195:0/3846598623 conn(0x7fa2fc0f0b70 msgr2=0x7fa2fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 shutdown_connections 2024-02-20T00:34:36.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:36.894+0000 7fa300aa7700 1 -- 172.21.15.195:0/3846598623 wait complete. 2024-02-20T00:34:36.896 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:37.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:36.892504+0000 mon.smithi087 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.195:0/3846598623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:38.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:36.607387+0000 mgr.smithi087.iydsqo (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:38.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:38.633 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:40.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- 172.21.15.195:0/4091746354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 msgr2=0x7f18c4069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:40.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 --2- 172.21.15.195:0/4091746354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c4069e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f18c0009fb0 tx=0x7f18c0009b30).stop 2024-02-20T00:34:40.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- 172.21.15.195:0/4091746354 shutdown_connections 2024-02-20T00:34:40.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 --2- 172.21.15.195:0/4091746354 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c4069e40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:40.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- 172.21.15.195:0/4091746354 >> 172.21.15.195:0/4091746354 conn(0x7f18c40f1ac0 msgr2=0x7f18c40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- 172.21.15.195:0/4091746354 shutdown_connections 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- 172.21.15.195:0/4091746354 wait complete. 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 Processor -- start 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.483+0000 7f18cad6c700 1 -- start start 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18cad6c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18cad6c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4108cc0 con 0x7f18c406b8a0 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18c9d6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18c9d6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53966/0 (socket says 172.21.15.195:53966) 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18c9d6a700 1 -- 172.21.15.195:0/1906335233 learned_addr learned my addr 172.21.15.195:0/1906335233 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18c9d6a700 1 -- 172.21.15.195:0/1906335233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c0005040 con 0x7f18c406b8a0 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18c9d6a700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f18c0005010 tx=0x7f18c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c0009320 con 0x7f18c406b8a0 2024-02-20T00:34:40.487 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c4108ec0 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18c0013380 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c0014e50 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.484+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c4109350 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.485+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18c00095f0 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.485+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4045ad0 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.485+0000 7f18ba7fc700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 0x7f18b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.485+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18c0040e60 con 0x7f18c406b8a0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.486+0000 7f18c9569700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 0x7f18b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:40.488 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.486+0000 7f18c9569700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 0x7f18b00323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f18b40096a0 tx=0x7f18b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:40.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.490+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18c0014420 con 0x7f18c406b8a0 2024-02-20T00:34:40.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:38.607673+0000 mgr.smithi087.iydsqo (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:40.696 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.696+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18c410a200 con 0x7f18c406b8a0 2024-02-20T00:34:40.697 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.696+0000 7f18ba7fc700 1 -- 172.21.15.195:0/1906335233 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18c001d980 con 0x7f18c406b8a0 2024-02-20T00:34:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:40.697 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 msgr2=0x7f18b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 0x7f18b00323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f18b40096a0 tx=0x7f18b4006b40).stop 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 msgr2=0x7f18c41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f18c0005010 tx=0x7f18c000b040).stop 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 shutdown_connections 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f18b002ff10 0x7f18b00323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 --2- 172.21.15.195:0/1906335233 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18c406b8a0 0x7f18c41087d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 >> 172.21.15.195:0/1906335233 conn(0x7f18c40f1ac0 msgr2=0x7f18c40f2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 shutdown_connections 2024-02-20T00:34:40.699 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:40.698+0000 7f18cad6c700 1 -- 172.21.15.195:0/1906335233 wait complete. 2024-02-20T00:34:40.700 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:41.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:40.696673+0000 mon.smithi087 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.195:0/1906335233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:42.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:42.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:42.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:40.608001+0000 mgr.smithi087.iydsqo (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:44.339 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- 172.21.15.195:0/2868312960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 msgr2=0x7effb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:44.339 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 --2- 172.21.15.195:0/2868312960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb40f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7effa8009fb0 tx=0x7effa8009b30).stop 2024-02-20T00:34:44.339 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- 172.21.15.195:0/2868312960 shutdown_connections 2024-02-20T00:34:44.339 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 --2- 172.21.15.195:0/2868312960 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb40f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- 172.21.15.195:0/2868312960 >> 172.21.15.195:0/2868312960 conn(0x7effb40f0b70 msgr2=0x7effb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- 172.21.15.195:0/2868312960 shutdown_connections 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- 172.21.15.195:0/2868312960 wait complete. 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 Processor -- start 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.336+0000 7effbaadf700 1 -- start start 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effbaadf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effbaadf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4106ca0 con 0x7effb40f5410 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effb887b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effb887b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:53984/0 (socket says 172.21.15.195:53984) 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effb887b700 1 -- 172.21.15.195:0/1542999094 learned_addr learned my addr 172.21.15.195:0/1542999094 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effb887b700 1 -- 172.21.15.195:0/1542999094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8005040 con 0x7effb40f5410 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.337+0000 7effb887b700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7effa8005010 tx=0x7effa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:44.340 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effa8009320 con 0x7effb40f5410 2024-02-20T00:34:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb4106ea0 con 0x7effb40f5410 2024-02-20T00:34:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effa8013380 con 0x7effb40f5410 2024-02-20T00:34:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effa8014e50 con 0x7effb40f5410 2024-02-20T00:34:44.341 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb4107330 con 0x7effb40f5410 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.338+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effa80095f0 con 0x7effb40f5410 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.339+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb4045ad0 con 0x7effb40f5410 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.339+0000 7effb17fa700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 0x7effa40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.339+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effa8040e90 con 0x7effb40f5410 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.339+0000 7effb3fff700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 0x7effa40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:44.342 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.339+0000 7effb3fff700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 0x7effa40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7effa00096a0 tx=0x7effa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:44.345 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.344+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effa8014420 con 0x7effb40f5410 2024-02-20T00:34:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:42.608389+0000 mgr.smithi087.iydsqo (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:44.547 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.546+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effb41081e0 con 0x7effb40f5410 2024-02-20T00:34:44.548 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.546+0000 7effb17fa700 1 -- 172.21.15.195:0/1542999094 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effa8010230 con 0x7effb40f5410 2024-02-20T00:34:44.548 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:44.548 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 msgr2=0x7effa40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 0x7effa40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7effa00096a0 tx=0x7effa0006b40).stop 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 msgr2=0x7effb41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7effa8005010 tx=0x7effa800b040).stop 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 shutdown_connections 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7effa402ff10 0x7effa40323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 --2- 172.21.15.195:0/1542999094 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7effb40f5410 0x7effb41067b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 >> 172.21.15.195:0/1542999094 conn(0x7effb40f0b70 msgr2=0x7effb405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 shutdown_connections 2024-02-20T00:34:44.550 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:44.548+0000 7effbaadf700 1 -- 172.21.15.195:0/1542999094 wait complete. 2024-02-20T00:34:44.551 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:45.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:45 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:44.547575+0000 mon.smithi087 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.195:0/1542999094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:46.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:46.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:46.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:44.608688+0000 mgr.smithi087.iydsqo (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:48.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.166+0000 7f95fb59e700 1 -- 172.21.15.195:0/4284014637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 msgr2=0x7f95fc0f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:48.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.166+0000 7f95fb59e700 1 --2- 172.21.15.195:0/4284014637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f4ac0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f95ec009fb0 tx=0x7f95ec009b30).stop 2024-02-20T00:34:48.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- 172.21.15.195:0/4284014637 shutdown_connections 2024-02-20T00:34:48.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 --2- 172.21.15.195:0/4284014637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f4ac0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:48.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- 172.21.15.195:0/4284014637 >> 172.21.15.195:0/4284014637 conn(0x7f95fc0f0120 msgr2=0x7f95fc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- 172.21.15.195:0/4284014637 shutdown_connections 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- 172.21.15.195:0/4284014637 wait complete. 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 Processor -- start 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- start start 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.167+0000 7f95fb59e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc0f5710 con 0x7f95fc0f46a0 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47244/0 (socket says 172.21.15.195:47244) 2024-02-20T00:34:48.170 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fa59c700 1 -- 172.21.15.195:0/3746378609 learned_addr learned my addr 172.21.15.195:0/3746378609 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fa59c700 1 -- 172.21.15.195:0/3746378609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec005040 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fa59c700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f95ec0089f0 tx=0x7f95ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ec009320 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc0f58b0 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95ec013380 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95ec014e20 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.168+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc0f5d40 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.169+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95ec0263a0 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.169+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc059100 con 0x7f95fc0f46a0 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.169+0000 7f95f2ffd700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 0x7f95e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:48.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.169+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95ec041220 con 0x7f95fc0f46a0 2024-02-20T00:34:48.173 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.169+0000 7f95f9d9b700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 0x7f95e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:48.173 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.170+0000 7f95f9d9b700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 0x7f95e80323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f95e40096a0 tx=0x7f95e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:48.175 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.173+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95ec01cb10 con 0x7f95fc0f46a0 2024-02-20T00:34:48.378 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.376+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95fc045ad0 con 0x7f95fc0f46a0 2024-02-20T00:34:48.379 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.377+0000 7f95f2ffd700 1 -- 172.21.15.195:0/3746378609 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95ec01cb10 con 0x7f95fc0f46a0 2024-02-20T00:34:48.379 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:48.379 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:48.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 msgr2=0x7f95e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:48.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 0x7f95e80323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f95e40096a0 tx=0x7f95e4006b40).stop 2024-02-20T00:34:48.380 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 msgr2=0x7f95fc0f7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f95ec0089f0 tx=0x7f95ec00b040).stop 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 shutdown_connections 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f95e802ff10 0x7f95e80323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 --2- 172.21.15.195:0/3746378609 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95fc0f46a0 0x7f95fc0f7290 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 >> 172.21.15.195:0/3746378609 conn(0x7f95fc0f0120 msgr2=0x7f95fc0f0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 shutdown_connections 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:48.379+0000 7f95fb59e700 1 -- 172.21.15.195:0/3746378609 wait complete. 2024-02-20T00:34:48.381 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:46.609028+0000 mgr.smithi087.iydsqo (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:49.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:48.378376+0000 mon.smithi087 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.195:0/3746378609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:50.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:50.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:50.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:48.609309+0000 mgr.smithi087.iydsqo (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:51.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.957+0000 7f7b805c2700 1 -- 172.21.15.195:0/3002977170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 msgr2=0x7f7b780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.957+0000 7f7b805c2700 1 --2- 172.21.15.195:0/3002977170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b780f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f7b68009fb0 tx=0x7f7b68009b30).stop 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- 172.21.15.195:0/3002977170 shutdown_connections 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 --2- 172.21.15.195:0/3002977170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b780f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- 172.21.15.195:0/3002977170 >> 172.21.15.195:0/3002977170 conn(0x7f7b780f1c70 msgr2=0x7f7b780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- 172.21.15.195:0/3002977170 shutdown_connections 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- 172.21.15.195:0/3002977170 wait complete. 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 Processor -- start 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- start start 2024-02-20T00:34:51.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b805c2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78106cd0 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b7e35e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b7e35e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47274/0 (socket says 172.21.15.195:47274) 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.958+0000 7f7b7e35e700 1 -- 172.21.15.195:0/1129846659 learned_addr learned my addr 172.21.15.195:0/1129846659 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b7e35e700 1 -- 172.21.15.195:0/1129846659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68005040 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b7e35e700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7b680089f0 tx=0x7f7b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b68009320 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b78106ed0 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b68013380 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b6801ed80 con 0x7f7b780f6510 2024-02-20T00:34:51.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.959+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78107360 con 0x7f7b780f6510 2024-02-20T00:34:51.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.960+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b6801e420 con 0x7f7b780f6510 2024-02-20T00:34:51.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.960+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b78045ad0 con 0x7f7b780f6510 2024-02-20T00:34:51.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.960+0000 7f7b6effd700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 0x7f7b64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:51.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.960+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b68041110 con 0x7f7b780f6510 2024-02-20T00:34:51.964 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.960+0000 7f7b7db5d700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 0x7f7b64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:51.964 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.961+0000 7f7b7db5d700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 0x7f7b64032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7b740096a0 tx=0x7f7b74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:51.966 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:51.965+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b680095f0 con 0x7f7b780f6510 2024-02-20T00:34:52.173 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.171+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b78108210 con 0x7f7b780f6510 2024-02-20T00:34:52.174 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.172+0000 7f7b6effd700 1 -- 172.21.15.195:0/1129846659 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b68004490 con 0x7f7b780f6510 2024-02-20T00:34:52.174 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:52.174 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:52.175 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.173+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 msgr2=0x7f7b64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.173+0000 7f7b805c2700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 0x7f7b64032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f7b740096a0 tx=0x7f7b74006b40).stop 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.173+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 msgr2=0x7f7b781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.173+0000 7f7b805c2700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f7b680089f0 tx=0x7f7b6800b040).stop 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 shutdown_connections 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7b6402ff60 0x7f7b64032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 --2- 172.21.15.195:0/1129846659 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7b780f6510 0x7f7b781067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 >> 172.21.15.195:0/1129846659 conn(0x7f7b780f1c70 msgr2=0x7f7b780f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 shutdown_connections 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:52.174+0000 7f7b805c2700 1 -- 172.21.15.195:0/1129846659 wait complete. 2024-02-20T00:34:52.176 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:50.609654+0000 mgr.smithi087.iydsqo (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:52.173241+0000 mon.smithi087 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.195:0/1129846659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:53.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:53.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:54.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:52.610039+0000 mgr.smithi087.iydsqo (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:55.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.946+0000 7f25daffd700 1 -- 172.21.15.195:0/3822226414 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d0013990 con 0x7f25dc0f5410 2024-02-20T00:34:55.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 -- 172.21.15.195:0/3822226414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc0f5410 msgr2=0x7f25dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:55.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 --2- 172.21.15.195:0/3822226414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc0f5410 0x7f25dc0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f25d0009fb0 tx=0x7f25d0009b30).stop 2024-02-20T00:34:55.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 -- 172.21.15.195:0/3822226414 shutdown_connections 2024-02-20T00:34:55.952 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 --2- 172.21.15.195:0/3822226414 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc0f5410 0x7f25dc0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 -- 172.21.15.195:0/3822226414 >> 172.21.15.195:0/3822226414 conn(0x7f25dc0f0b70 msgr2=0x7f25dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 -- 172.21.15.195:0/3822226414 shutdown_connections 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 -- 172.21.15.195:0/3822226414 wait complete. 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.947+0000 7f25e22b4700 1 Processor -- start 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25e22b4700 1 -- start start 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25e22b4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25e22b4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc109410 con 0x7f25dc108b00 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25dbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25dbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:47296/0 (socket says 172.21.15.195:47296) 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25dbfff700 1 -- 172.21.15.195:0/4263984680 learned_addr learned my addr 172.21.15.195:0/4263984680 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25dbfff700 1 -- 172.21.15.195:0/4263984680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0005040 con 0x7f25dc108b00 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.948+0000 7f25dbfff700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f25d0009f80 tx=0x7f25d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d0013380 con 0x7f25dc108b00 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc109610 con 0x7f25dc108b00 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25d0013790 con 0x7f25dc108b00 2024-02-20T00:34:55.953 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d0014420 con 0x7f25dc108b00 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc10a160 con 0x7f25dc108b00 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.949+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25d0025060 con 0x7f25dc108b00 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.950+0000 7f25d8ff9700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.950+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25d0040dc0 con 0x7f25dc108b00 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.950+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc059100 con 0x7f25dc108b00 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.950+0000 7f25db7fe700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:55.954 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.950+0000 7f25db7fe700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 0x7f25c4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f25cc0096a0 tx=0x7f25cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:55.956 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:55.954+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25d0028440 con 0x7f25dc108b00 2024-02-20T00:34:56.162 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.161+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25dc10a880 con 0x7f25dc108b00 2024-02-20T00:34:56.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.161+0000 7f25d8ff9700 1 -- 172.21.15.195:0/4263984680 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25d001d3a0 con 0x7f25dc108b00 2024-02-20T00:34:56.163 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:34:56.163 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 msgr2=0x7f25c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 0x7f25c4032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f25cc0096a0 tx=0x7f25cc006b40).stop 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 msgr2=0x7f25dc108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f25d0009f80 tx=0x7f25d0009b30).stop 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 shutdown_connections 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f25c402ff60 0x7f25c4032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 --2- 172.21.15.195:0/4263984680 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f25dc108b00 0x7f25dc108f20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 >> 172.21.15.195:0/4263984680 conn(0x7f25dc0f0b70 msgr2=0x7f25dc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 shutdown_connections 2024-02-20T00:34:56.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:56.163+0000 7f25e22b4700 1 -- 172.21.15.195:0/4263984680 wait complete. 2024-02-20T00:34:56.166 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:34:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:54.610329+0000 mgr.smithi087.iydsqo (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:56.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:56.162647+0000 mon.smithi087 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.195:0/4263984680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:34:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:56.673377+0000 mon.smithi087 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:34:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:34:56.673569+0000 mon.smithi087 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:34:57.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:34:57.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:34:58.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:34:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:56.610691+0000 mgr.smithi087.iydsqo (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:34:59.884 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- 172.21.15.195:0/550343727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 msgr2=0x7f19700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 --2- 172.21.15.195:0/550343727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19700f66c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1960009fb0 tx=0x7f1960009b30).stop 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- 172.21.15.195:0/550343727 shutdown_connections 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 --2- 172.21.15.195:0/550343727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19700f66c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- 172.21.15.195:0/550343727 >> 172.21.15.195:0/550343727 conn(0x7f19700f1ac0 msgr2=0x7f19700f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- 172.21.15.195:0/550343727 shutdown_connections 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- 172.21.15.195:0/550343727 wait complete. 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 Processor -- start 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.881+0000 7f19761dc700 1 -- start start 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19761dc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19761dc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970106ab0 con 0x7f19700f62a0 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19751da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19751da700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40374/0 (socket says 172.21.15.195:40374) 2024-02-20T00:34:59.885 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19751da700 1 -- 172.21.15.195:0/1340498976 learned_addr learned my addr 172.21.15.195:0/1340498976 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19751da700 1 -- 172.21.15.195:0/1340498976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960005040 con 0x7f19700f62a0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.882+0000 7f19751da700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f19600089f0 tx=0x7f196000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.883+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1960009320 con 0x7f19700f62a0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.883+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1970106cb0 con 0x7f19700f62a0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.883+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1960013380 con 0x7f19700f62a0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.883+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1960014e20 con 0x7f19700f62a0 2024-02-20T00:34:59.886 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.883+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1970107140 con 0x7f19700f62a0 2024-02-20T00:34:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19600095f0 con 0x7f19700f62a0 2024-02-20T00:34:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970045ad0 con 0x7f19700f62a0 2024-02-20T00:34:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f1965ffb700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 0x7f195c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:34:59.887 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1960040ec0 con 0x7f19700f62a0 2024-02-20T00:34:59.888 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f19749d9700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 0x7f195c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:34:59.888 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.884+0000 7f19749d9700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 0x7f195c0323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f196c0096a0 tx=0x7f196c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:34:59.890 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:34:59.889+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1960014420 con 0x7f19700f62a0 2024-02-20T00:35:00.100 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.097+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1970107f90 con 0x7f19700f62a0 2024-02-20T00:35:00.100 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.098+0000 7f1965ffb700 1 -- 172.21.15.195:0/1340498976 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1960010230 con 0x7f19700f62a0 2024-02-20T00:35:00.101 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:00.101 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:00.102 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 msgr2=0x7f195c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:00.102 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 0x7f195c0323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f196c0096a0 tx=0x7f196c006b40).stop 2024-02-20T00:35:00.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 msgr2=0x7f19701065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:00.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f19600089f0 tx=0x7f196000b040).stop 2024-02-20T00:35:00.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 shutdown_connections 2024-02-20T00:35:00.103 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f195c02ff10 0x7f195c0323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 --2- 172.21.15.195:0/1340498976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f19700f62a0 0x7f19701065c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 >> 172.21.15.195:0/1340498976 conn(0x7f19700f1ac0 msgr2=0x7f19700f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 shutdown_connections 2024-02-20T00:35:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:00.100+0000 7f19761dc700 1 -- 172.21.15.195:0/1340498976 wait complete. 2024-02-20T00:35:00.104 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:34:58.611002+0000 mgr.smithi087.iydsqo (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:00.099064+0000 mon.smithi087 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.195:0/1340498976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:01.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:01.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:02.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:00.611356+0000 mgr.smithi087.iydsqo (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 -- 172.21.15.195:0/2895902039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 msgr2=0x7fbc2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/2895902039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c0f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fbc20009fb0 tx=0x7fbc20009b30).stop 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 -- 172.21.15.195:0/2895902039 shutdown_connections 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/2895902039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c0f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 -- 172.21.15.195:0/2895902039 >> 172.21.15.195:0/2895902039 conn(0x7fbc2c0f09a0 msgr2=0x7fbc2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:03.428 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 -- 172.21.15.195:0/2895902039 shutdown_connections 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.424+0000 7fbc2bfff700 1 -- 172.21.15.195:0/2895902039 wait complete. 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2bfff700 1 Processor -- start 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2bfff700 1 -- start start 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2bfff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c1059c0 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:40384/0 (socket says 172.21.15.195:40384) 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2affd700 1 -- 172.21.15.195:0/1958913403 learned_addr learned my addr 172.21.15.195:0/1958913403 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.425+0000 7fbc2affd700 1 -- 172.21.15.195:0/1958913403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20005040 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc2affd700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbc200089f0 tx=0x7fbc2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc20009320 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c105bc0 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c106050 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.429 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc20013380 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.430 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.426+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc20014db0 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.430 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c004fa0 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.430 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc200095f0 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc137fe700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 0x7fbc14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:03.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc20040e70 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc2a7fc700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 0x7fbc14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:03.431 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.427+0000 7fbc2a7fc700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 0x7fbc14032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbc1c0096a0 tx=0x7fbc1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:03.433 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.431+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc20004490 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.635+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc0c005670 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.637 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.636+0000 7fbc137fe700 1 -- 172.21.15.195:0/1958913403 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc20012330 con 0x7fbc2c0f51a0 2024-02-20T00:35:03.638 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:03.638 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:03.639 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 msgr2=0x7fbc14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 0x7fbc14032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbc1c0096a0 tx=0x7fbc1c006b40).stop 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 msgr2=0x7fbc2c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbc200089f0 tx=0x7fbc2000b040).stop 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 shutdown_connections 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fbc1402ff60 0x7fbc14032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 --2- 172.21.15.195:0/1958913403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc2c0f51a0 0x7fbc2c1054d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 >> 172.21.15.195:0/1958913403 conn(0x7fbc2c0f09a0 msgr2=0x7fbc2c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 shutdown_connections 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:03.637+0000 7fbc2bfff700 1 -- 172.21.15.195:0/1958913403 wait complete. 2024-02-20T00:35:03.640 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:04.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:02.611658+0000 mgr.smithi087.iydsqo (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:04.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:03.637122+0000 mon.smithi087 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.195:0/1958913403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:05.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:05.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:06.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:04.611987+0000 mgr.smithi087.iydsqo (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:07.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/4115899876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 msgr2=0x7fb2a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:07.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/4115899876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a40f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb294009fb0 tx=0x7fb294009b30).stop 2024-02-20T00:35:07.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/4115899876 shutdown_connections 2024-02-20T00:35:07.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/4115899876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a40f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/4115899876 >> 172.21.15.195:0/4115899876 conn(0x7fb2a40f1c70 msgr2=0x7fb2a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/4115899876 shutdown_connections 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.187+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/4115899876 wait complete. 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2ac3d2700 1 Processor -- start 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2ac3d2700 1 -- start start 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2ac3d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2ac3d2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a4069e20 con 0x7fb2a40f6510 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2aa16e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2aa16e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57304/0 (socket says 172.21.15.195:57304) 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2aa16e700 1 -- 172.21.15.195:0/893925862 learned_addr learned my addr 172.21.15.195:0/893925862 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2aa16e700 1 -- 172.21.15.195:0/893925862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294005040 con 0x7fb2a40f6510 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.188+0000 7fb2aa16e700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb294005010 tx=0x7fb29400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:07.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb294009320 con 0x7fb2a40f6510 2024-02-20T00:35:07.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a4069fc0 con 0x7fb2a40f6510 2024-02-20T00:35:07.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb294013380 con 0x7fb2a40f6510 2024-02-20T00:35:07.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb294014e50 con 0x7fb2a40f6510 2024-02-20T00:35:07.192 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a406a450 con 0x7fb2a40f6510 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a4059100 con 0x7fb2a40f6510 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.189+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2940095f0 con 0x7fb2a40f6510 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.190+0000 7fb29affd700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 0x7fb2900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.190+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb294040f10 con 0x7fb2a40f6510 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.190+0000 7fb2a996d700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 0x7fb2900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:07.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.190+0000 7fb2a996d700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 0x7fb2900323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb2a00096a0 tx=0x7fb2a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:07.196 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.194+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb294014420 con 0x7fb2a40f6510 2024-02-20T00:35:07.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.401+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2a40fc0a0 con 0x7fb2a40f6510 2024-02-20T00:35:07.403 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.402+0000 7fb29affd700 1 -- 172.21.15.195:0/893925862 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb294010230 con 0x7fb2a40f6510 2024-02-20T00:35:07.403 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:07.403 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:07.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 msgr2=0x7fb2900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 0x7fb2900323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb2a00096a0 tx=0x7fb2a0006b40).stop 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 msgr2=0x7fb2a406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb294005010 tx=0x7fb29400b040).stop 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 shutdown_connections 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb29002ff10 0x7fb2900323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 --2- 172.21.15.195:0/893925862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a40f6510 0x7fb2a406b9a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 >> 172.21.15.195:0/893925862 conn(0x7fb2a40f1c70 msgr2=0x7fb2a40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 shutdown_connections 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:07.403+0000 7fb2ac3d2700 1 -- 172.21.15.195:0/893925862 wait complete. 2024-02-20T00:35:07.406 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:08.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:06.612316+0000 mgr.smithi087.iydsqo (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:08.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:07.402887+0000 mon.smithi087 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.195:0/893925862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:09.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:09.158 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:10.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:08.612719+0000 mgr.smithi087.iydsqo (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:10.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 -- 172.21.15.195:0/3510209909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 msgr2=0x7ff5840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:10.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 --2- 172.21.15.195:0/3510209909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5840f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7ff574009fb0 tx=0x7ff574009b30).stop 2024-02-20T00:35:10.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 -- 172.21.15.195:0/3510209909 shutdown_connections 2024-02-20T00:35:10.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 --2- 172.21.15.195:0/3510209909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5840f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:10.960 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 -- 172.21.15.195:0/3510209909 >> 172.21.15.195:0/3510209909 conn(0x7ff5840f0b70 msgr2=0x7ff5840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 -- 172.21.15.195:0/3510209909 shutdown_connections 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.956+0000 7ff58a544700 1 -- 172.21.15.195:0/3510209909 wait complete. 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.957+0000 7ff58a544700 1 Processor -- start 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.957+0000 7ff58a544700 1 -- start start 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.957+0000 7ff58a544700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.957+0000 7ff58a544700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584105bd0 con 0x7ff5840f5410 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57314/0 (socket says 172.21.15.195:57314) 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff583fff700 1 -- 172.21.15.195:0/4057063566 learned_addr learned my addr 172.21.15.195:0/4057063566 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff583fff700 1 -- 172.21.15.195:0/4057063566 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574005040 con 0x7ff5840f5410 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff583fff700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff574005010 tx=0x7ff57400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff574009320 con 0x7ff5840f5410 2024-02-20T00:35:10.961 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff584105dd0 con 0x7ff5840f5410 2024-02-20T00:35:10.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff574013380 con 0x7ff5840f5410 2024-02-20T00:35:10.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff574014e50 con 0x7ff5840f5410 2024-02-20T00:35:10.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.958+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff584106260 con 0x7ff5840f5410 2024-02-20T00:35:10.962 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.959+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff574014420 con 0x7ff5840f5410 2024-02-20T00:35:10.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.959+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff584045ad0 con 0x7ff5840f5410 2024-02-20T00:35:10.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.959+0000 7ff580ff9700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 0x7ff5700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:10.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.959+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff574040f40 con 0x7ff5840f5410 2024-02-20T00:35:10.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.960+0000 7ff5837fe700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 0x7ff5700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:10.963 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.960+0000 7ff5837fe700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 0x7ff5700323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff56c0096a0 tx=0x7ff56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:10.965 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:10.964+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5740095f0 con 0x7ff5840f5410 2024-02-20T00:35:11.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.167+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff584107110 con 0x7ff5840f5410 2024-02-20T00:35:11.169 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.168+0000 7ff580ff9700 1 -- 172.21.15.195:0/4057063566 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff57401db70 con 0x7ff5840f5410 2024-02-20T00:35:11.169 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:11.170 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 msgr2=0x7ff5700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 0x7ff5700323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff56c0096a0 tx=0x7ff56c006b40).stop 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 msgr2=0x7ff5841056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7ff574005010 tx=0x7ff57400b040).stop 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 shutdown_connections 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff57002ff10 0x7ff5700323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 --2- 172.21.15.195:0/4057063566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff5840f5410 0x7ff5841056e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:11.171 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 >> 172.21.15.195:0/4057063566 conn(0x7ff5840f0b70 msgr2=0x7ff5840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:11.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 shutdown_connections 2024-02-20T00:35:11.172 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:11.169+0000 7ff58a544700 1 -- 172.21.15.195:0/4057063566 wait complete. 2024-02-20T00:35:11.172 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:11.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:11.168992+0000 mon.smithi087 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.195:0/4057063566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:12.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:10.613081+0000 mgr.smithi087.iydsqo (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:12.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:12.814 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:14.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 -- 172.21.15.195:0/1919940107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 msgr2=0x7f93340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:14.736 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 --2- 172.21.15.195:0/1919940107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93340f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f9328009fb0 tx=0x7f9328009b30).stop 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 -- 172.21.15.195:0/1919940107 shutdown_connections 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 --2- 172.21.15.195:0/1919940107 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93340f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 -- 172.21.15.195:0/1919940107 >> 172.21.15.195:0/1919940107 conn(0x7f93340f0b70 msgr2=0x7f93340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 -- 172.21.15.195:0/1919940107 shutdown_connections 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.734+0000 7f933ad1f700 1 -- 172.21.15.195:0/1919940107 wait complete. 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f933ad1f700 1 Processor -- start 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f933ad1f700 1 -- start start 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f933ad1f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:14.737 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f933ad1f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334105bd0 con 0x7f93340f5410 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f9338abb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f9338abb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57318/0 (socket says 172.21.15.195:57318) 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.735+0000 7f9338abb700 1 -- 172.21.15.195:0/3412025847 learned_addr learned my addr 172.21.15.195:0/3412025847 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f9338abb700 1 -- 172.21.15.195:0/3412025847 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328005040 con 0x7f93340f5410 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f9338abb700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9328005010 tx=0x7f932800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9328009320 con 0x7f93340f5410 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9334105dd0 con 0x7f93340f5410 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9328013380 con 0x7f93340f5410 2024-02-20T00:35:14.739 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9328014e50 con 0x7f93340f5410 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.736+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9334106260 con 0x7f93340f5410 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9328014420 con 0x7f93340f5410 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9334045ad0 con 0x7f93340f5410 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f93317fa700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 0x7f93240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9328040f40 con 0x7f93340f5410 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f9333fff700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 0x7f93240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:14.740 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.737+0000 7f9333fff700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 0x7f93240323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f93200096a0 tx=0x7f9320006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:14.743 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.741+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93280095f0 con 0x7f93340f5410 2024-02-20T00:35:14.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:12.613393+0000 mgr.smithi087.iydsqo (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.945+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9334107110 con 0x7f93340f5410 2024-02-20T00:35:14.947 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.946+0000 7f93317fa700 1 -- 172.21.15.195:0/3412025847 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f932801db70 con 0x7f93340f5410 2024-02-20T00:35:14.948 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:14.948 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.947+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 msgr2=0x7f93240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.947+0000 7f933ad1f700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 0x7f93240323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f93200096a0 tx=0x7f9320006b40).stop 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.947+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 msgr2=0x7f93341056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.947+0000 7f933ad1f700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9328005010 tx=0x7f932800b040).stop 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 shutdown_connections 2024-02-20T00:35:14.949 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f932402ff10 0x7f93240323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 --2- 172.21.15.195:0/3412025847 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f93340f5410 0x7f93341056e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 >> 172.21.15.195:0/3412025847 conn(0x7f93340f0b70 msgr2=0x7f93340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 shutdown_connections 2024-02-20T00:35:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:14.948+0000 7f933ad1f700 1 -- 172.21.15.195:0/3412025847 wait complete. 2024-02-20T00:35:14.950 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:15.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:15 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:14.947121+0000 mon.smithi087 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.195:0/3412025847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:16.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:16.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:16.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:14.613752+0000 mgr.smithi087.iydsqo (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:18.455 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 -- 172.21.15.195:0/1165447229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 msgr2=0x7f86100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:18.455 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 --2- 172.21.15.195:0/1165447229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f86100f90f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f8604009fb0 tx=0x7f8604009b30).stop 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 -- 172.21.15.195:0/1165447229 shutdown_connections 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 --2- 172.21.15.195:0/1165447229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f86100f90f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 -- 172.21.15.195:0/1165447229 >> 172.21.15.195:0/1165447229 conn(0x7f86100f0b70 msgr2=0x7f86100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 -- 172.21.15.195:0/1165447229 shutdown_connections 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.452+0000 7f8614f28700 1 -- 172.21.15.195:0/1165447229 wait complete. 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f8614f28700 1 Processor -- start 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f8614f28700 1 -- start start 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f8614f28700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f8614f28700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8610106c70 con 0x7f8610060280 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f860e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:18.456 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f860e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57066/0 (socket says 172.21.15.195:57066) 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f860e59c700 1 -- 172.21.15.195:0/1413463284 learned_addr learned my addr 172.21.15.195:0/1413463284 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.453+0000 7f860e59c700 1 -- 172.21.15.195:0/1413463284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604005040 con 0x7f8610060280 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f860e59c700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f86040089f0 tx=0x7f860400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8604009320 con 0x7f8610060280 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8610106e70 con 0x7f8610060280 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8604013380 con 0x7f8610060280 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8604014e20 con 0x7f8610060280 2024-02-20T00:35:18.457 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.454+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8610107300 con 0x7f8610060280 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8610045ad0 con 0x7f8610060280 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8604014420 con 0x7f8610060280 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f85feffd700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 0x7f85f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8604040f50 con 0x7f8610060280 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f860dd9b700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 0x7f85f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:18.458 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.455+0000 7f860dd9b700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 0x7f85f80323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f86000096a0 tx=0x7f8600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:18.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.460+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f860406d2b0 con 0x7f8610060280 2024-02-20T00:35:18.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.665+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86101081b0 con 0x7f8610060280 2024-02-20T00:35:18.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.666+0000 7f85feffd700 1 -- 172.21.15.195:0/1413463284 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f86040095f0 con 0x7f8610060280 2024-02-20T00:35:18.667 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:18.667 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 msgr2=0x7f85f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 0x7f85f80323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f86000096a0 tx=0x7f8600006b40).stop 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 msgr2=0x7f8610106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f86040089f0 tx=0x7f860400b040).stop 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 shutdown_connections 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f85f802ff10 0x7f85f80323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:18.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 --2- 172.21.15.195:0/1413463284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8610060280 0x7f8610106780 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:18.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 >> 172.21.15.195:0/1413463284 conn(0x7f86100f0b70 msgr2=0x7f86100f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:18.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 shutdown_connections 2024-02-20T00:35:18.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:18.667+0000 7f8614f28700 1 -- 172.21.15.195:0/1413463284 wait complete. 2024-02-20T00:35:18.670 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:18.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:16.614019+0000 mgr.smithi087.iydsqo (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:19.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:19 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:18.666989+0000 mon.smithi087 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.195:0/1413463284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:20.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:20.442 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:18.614374+0000 mgr.smithi087.iydsqo (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 -- 172.21.15.195:0/2086045954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 msgr2=0x7f4af00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 --2- 172.21.15.195:0/2086045954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af00f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4ae0009fb0 tx=0x7f4ae0009b30).stop 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 -- 172.21.15.195:0/2086045954 shutdown_connections 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 --2- 172.21.15.195:0/2086045954 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af00f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 -- 172.21.15.195:0/2086045954 >> 172.21.15.195:0/2086045954 conn(0x7f4af00f0b70 msgr2=0x7f4af00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 -- 172.21.15.195:0/2086045954 shutdown_connections 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.461+0000 7f4af5af2700 1 -- 172.21.15.195:0/2086045954 wait complete. 2024-02-20T00:35:22.464 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.462+0000 7f4af5af2700 1 Processor -- start 2024-02-20T00:35:22.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.462+0000 7f4af5af2700 1 -- start start 2024-02-20T00:35:22.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4af5af2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:22.465 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4af5af2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0060220 con 0x7f4af00f5410 2024-02-20T00:35:22.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4aef7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:22.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4aef7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:57072/0 (socket says 172.21.15.195:57072) 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4aef7fe700 1 -- 172.21.15.195:0/3650165272 learned_addr learned my addr 172.21.15.195:0/3650165272 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4aef7fe700 1 -- 172.21.15.195:0/3650165272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae0005040 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.463+0000 7f4aef7fe700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4ae0005010 tx=0x7f4ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.464+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae0009320 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.464+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0060420 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.464+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae0013380 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.464+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae0014e50 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.464+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0109bb0 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ae00095f0 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af0059100 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4ad7fff700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 0x7f4ad80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ae0040e90 con 0x7f4af00f5410 2024-02-20T00:35:22.467 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4aeeffd700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 0x7f4ad80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:22.468 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.465+0000 7f4aeeffd700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 0x7f4ad80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:22.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.470+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ae0014420 con 0x7f4af00f5410 2024-02-20T00:35:22.678 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.677+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4af0045ad0 con 0x7f4af00f5410 2024-02-20T00:35:22.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.677+0000 7f4ad7fff700 1 -- 172.21.15.195:0/3650165272 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ae0010230 con 0x7f4af00f5410 2024-02-20T00:35:22.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:22.679 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:22.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 msgr2=0x7f4ad80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:22.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 0x7f4ad80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4ae40096a0 tx=0x7f4ae4006b40).stop 2024-02-20T00:35:22.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 msgr2=0x7f4af0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:22.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4ae0005010 tx=0x7f4ae000b040).stop 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 shutdown_connections 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4ad802ff10 0x7f4ad80323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 --2- 172.21.15.195:0/3650165272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4af00f5410 0x7f4af0109500 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 >> 172.21.15.195:0/3650165272 conn(0x7f4af00f0b70 msgr2=0x7f4af00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 shutdown_connections 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:22.679+0000 7f4af5af2700 1 -- 172.21.15.195:0/3650165272 wait complete. 2024-02-20T00:35:22.682 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:20.614731+0000 mgr.smithi087.iydsqo (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:23.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:23 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:22.678683+0000 mon.smithi087 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.195:0/3650165272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:24.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:24.479 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:24.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:22.615051+0000 mgr.smithi087.iydsqo (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 -- 172.21.15.195:0/2743206774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 msgr2=0x7f2c600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 --2- 172.21.15.195:0/2743206774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c600f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f2c5c009fb0 tx=0x7f2c5c009b30).stop 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 -- 172.21.15.195:0/2743206774 shutdown_connections 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 --2- 172.21.15.195:0/2743206774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c600f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 -- 172.21.15.195:0/2743206774 >> 172.21.15.195:0/2743206774 conn(0x7f2c600f1c70 msgr2=0x7f2c600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 -- 172.21.15.195:0/2743206774 shutdown_connections 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 -- 172.21.15.195:0/2743206774 wait complete. 2024-02-20T00:35:26.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.446+0000 7f2c68468700 1 Processor -- start 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c68468700 1 -- start start 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c68468700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c68468700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c60106cd0 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c66204700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c66204700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35350/0 (socket says 172.21.15.195:35350) 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c66204700 1 -- 172.21.15.195:0/3827322171 learned_addr learned my addr 172.21.15.195:0/3827322171 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.447+0000 7f2c66204700 1 -- 172.21.15.195:0/3827322171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c005040 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c66204700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2c5c005010 tx=0x7f2c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c5c009320 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c60106ed0 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c5c013380 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c5c014e50 con 0x7f2c600f6510 2024-02-20T00:35:26.451 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.448+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60107360 con 0x7f2c600f6510 2024-02-20T00:35:26.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.449+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c5c0263a0 con 0x7f2c600f6510 2024-02-20T00:35:26.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.449+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c60045ad0 con 0x7f2c600f6510 2024-02-20T00:35:26.452 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.449+0000 7f2c56ffd700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 0x7f2c4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:26.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.449+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c5c041270 con 0x7f2c600f6510 2024-02-20T00:35:26.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.449+0000 7f2c65a03700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 0x7f2c4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:26.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.450+0000 7f2c65a03700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 0x7f2c4c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:26.455 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.454+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c5c01d470 con 0x7f2c600f6510 2024-02-20T00:35:26.663 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.661+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c60108210 con 0x7f2c600f6510 2024-02-20T00:35:26.663 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.663+0000 7f2c56ffd700 1 -- 172.21.15.195:0/3827322171 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c5c012330 con 0x7f2c600f6510 2024-02-20T00:35:26.664 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:26.664 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.663+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 msgr2=0x7f2c4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.663+0000 7f2c68468700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 0x7f2c4c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).stop 2024-02-20T00:35:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.663+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 msgr2=0x7f2c601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.663+0000 7f2c68468700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f2c5c005010 tx=0x7f2c5c00b040).stop 2024-02-20T00:35:26.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 shutdown_connections 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f2c4c02ff10 0x7f2c4c0323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 --2- 172.21.15.195:0/3827322171 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c600f6510 0x7f2c601067e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 >> 172.21.15.195:0/3827322171 conn(0x7f2c600f1c70 msgr2=0x7f2c600f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 shutdown_connections 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:26.664+0000 7f2c68468700 1 -- 172.21.15.195:0/3827322171 wait complete. 2024-02-20T00:35:26.666 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:26.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:24.615420+0000 mgr.smithi087.iydsqo (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:27.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:27 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:26.663203+0000 mon.smithi087 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.195:0/3827322171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:28.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:28.361 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:28.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:26.615726+0000 mgr.smithi087.iydsqo (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:30.190 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.188+0000 7f562728b700 1 -- 172.21.15.195:0/940851030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 msgr2=0x7f56200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.188+0000 7f562728b700 1 --2- 172.21.15.195:0/940851030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f5610009fb0 tx=0x7f5610009b30).stop 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.188+0000 7f562728b700 1 -- 172.21.15.195:0/940851030 shutdown_connections 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 --2- 172.21.15.195:0/940851030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 -- 172.21.15.195:0/940851030 >> 172.21.15.195:0/940851030 conn(0x7f56200f0b70 msgr2=0x7f56200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 -- 172.21.15.195:0/940851030 shutdown_connections 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 -- 172.21.15.195:0/940851030 wait complete. 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 Processor -- start 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.189+0000 7f562728b700 1 -- start start 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f562728b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f562728b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200fb960 con 0x7f56200f5410 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f5625027700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f5625027700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35360/0 (socket says 172.21.15.195:35360) 2024-02-20T00:35:30.191 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f5625027700 1 -- 172.21.15.195:0/2596424413 learned_addr learned my addr 172.21.15.195:0/2596424413 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f5625027700 1 -- 172.21.15.195:0/2596424413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610005040 con 0x7f56200f5410 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.190+0000 7f5625027700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5610005010 tx=0x7f561000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.191+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5610009320 con 0x7f56200f5410 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.191+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56200fbb00 con 0x7f56200f5410 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.191+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5610013380 con 0x7f56200f5410 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.191+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5610014e50 con 0x7f56200f5410 2024-02-20T00:35:30.193 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.191+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56200fbf90 con 0x7f56200f5410 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56100095f0 con 0x7f56200f5410 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5620059100 con 0x7f56200f5410 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f5615ffb700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 0x7f560c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5610040f80 con 0x7f56200f5410 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f5624826700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 0x7f560c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:30.194 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.192+0000 7f5624826700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 0x7f560c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f561c0096a0 tx=0x7f561c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.197+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5610014420 con 0x7f56200f5410 2024-02-20T00:35:30.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.400+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56200fafa0 con 0x7f56200f5410 2024-02-20T00:35:30.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.401+0000 7f5615ffb700 1 -- 172.21.15.195:0/2596424413 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5610010230 con 0x7f56200f5410 2024-02-20T00:35:30.401 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:30.401 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:30.403 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 msgr2=0x7f560c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:30.403 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 0x7f560c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f561c0096a0 tx=0x7f561c006b40).stop 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 msgr2=0x7f56200fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5610005010 tx=0x7f561000b040).stop 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 shutdown_connections 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f560c02ff10 0x7f560c0323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 --2- 172.21.15.195:0/2596424413 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56200f5410 0x7f56200fd4e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 >> 172.21.15.195:0/2596424413 conn(0x7f56200f0b70 msgr2=0x7f562005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 shutdown_connections 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:30.402+0000 7f562728b700 1 -- 172.21.15.195:0/2596424413 wait complete. 2024-02-20T00:35:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:28.616006+0000 mgr.smithi087.iydsqo (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:31.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:31 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:30.400994+0000 mon.smithi087 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.195:0/2596424413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:32.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:32.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:32.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:30.616305+0000 mgr.smithi087.iydsqo (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.032+0000 7f062fabb700 1 -- 172.21.15.195:0/40442732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 msgr2=0x7f06280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.032+0000 7f062fabb700 1 --2- 172.21.15.195:0/40442732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06280f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f061c009fb0 tx=0x7f061c009b30).stop 2024-02-20T00:35:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.032+0000 7f062fabb700 1 -- 172.21.15.195:0/40442732 shutdown_connections 2024-02-20T00:35:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.032+0000 7f062fabb700 1 --2- 172.21.15.195:0/40442732 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06280f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:34.036 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.032+0000 7f062fabb700 1 -- 172.21.15.195:0/40442732 >> 172.21.15.195:0/40442732 conn(0x7f06280f0b70 msgr2=0x7f06280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 -- 172.21.15.195:0/40442732 shutdown_connections 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 -- 172.21.15.195:0/40442732 wait complete. 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 Processor -- start 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 -- start start 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062fabb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0628106cd0 con 0x7f06280f5410 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062d857700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062d857700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35380/0 (socket says 172.21.15.195:35380) 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.033+0000 7f062d857700 1 -- 172.21.15.195:0/3694612887 learned_addr learned my addr 172.21.15.195:0/3694612887 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f062d857700 1 -- 172.21.15.195:0/3694612887 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c005040 con 0x7f06280f5410 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f062d857700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f061c005010 tx=0x7f061c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f061c009320 con 0x7f06280f5410 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0628106ed0 con 0x7f06280f5410 2024-02-20T00:35:34.037 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f061c013380 con 0x7f06280f5410 2024-02-20T00:35:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f061c014e50 con 0x7f06280f5410 2024-02-20T00:35:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.034+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0628107360 con 0x7f06280f5410 2024-02-20T00:35:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f061c0263a0 con 0x7f06280f5410 2024-02-20T00:35:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f061a7fc700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 0x7f06140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:34.038 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0628045ad0 con 0x7f06280f5410 2024-02-20T00:35:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f061c041200 con 0x7f06280f5410 2024-02-20T00:35:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f062d056700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 0x7f06140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:34.039 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.035+0000 7f062d056700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 0x7f06140323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f06240096a0 tx=0x7f0624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:34.041 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.040+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f061c012920 con 0x7f06280f5410 2024-02-20T00:35:34.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.246+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06280fb090 con 0x7f06280f5410 2024-02-20T00:35:34.248 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.247+0000 7f061a7fc700 1 -- 172.21.15.195:0/3694612887 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f061c012330 con 0x7f06280f5410 2024-02-20T00:35:34.249 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:34.249 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:34.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 msgr2=0x7f06140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:34.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 0x7f06140323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f06240096a0 tx=0x7f0624006b40).stop 2024-02-20T00:35:34.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 msgr2=0x7f06281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:34.250 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f061c005010 tx=0x7f061c00b040).stop 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 shutdown_connections 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f061402ff10 0x7f06140323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 --2- 172.21.15.195:0/3694612887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06280f5410 0x7f06281067e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 >> 172.21.15.195:0/3694612887 conn(0x7f06280f0b70 msgr2=0x7f062805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 shutdown_connections 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:34.249+0000 7f062fabb700 1 -- 172.21.15.195:0/3694612887 wait complete. 2024-02-20T00:35:34.251 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:34.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:32.616562+0000 mgr.smithi087.iydsqo (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:34.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:34.248154+0000 mon.smithi087 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.195:0/3694612887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:35.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:35.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:36.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:34.616911+0000 mgr.smithi087.iydsqo (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:37.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 -- 172.21.15.195:0/510571784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 msgr2=0x7f13440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:37.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 --2- 172.21.15.195:0/510571784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13440f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f1338009fb0 tx=0x7f1338009b30).stop 2024-02-20T00:35:37.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 -- 172.21.15.195:0/510571784 shutdown_connections 2024-02-20T00:35:37.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 --2- 172.21.15.195:0/510571784 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13440f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 -- 172.21.15.195:0/510571784 >> 172.21.15.195:0/510571784 conn(0x7f13440f0b70 msgr2=0x7f13440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 -- 172.21.15.195:0/510571784 shutdown_connections 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.896+0000 7f134ac8b700 1 -- 172.21.15.195:0/510571784 wait complete. 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f134ac8b700 1 Processor -- start 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f134ac8b700 1 -- start start 2024-02-20T00:35:37.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f134ac8b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:37.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f134ac8b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1344060280 con 0x7f13440f5410 2024-02-20T00:35:37.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f1348a27700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:37.904 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f1348a27700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60610/0 (socket says 172.21.15.195:60610) 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.897+0000 7f1348a27700 1 -- 172.21.15.195:0/1807756612 learned_addr learned my addr 172.21.15.195:0/1807756612 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f1348a27700 1 -- 172.21.15.195:0/1807756612 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338005040 con 0x7f13440f5410 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f1348a27700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1338005010 tx=0x7f133800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1338009320 con 0x7f13440f5410 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1344060480 con 0x7f13440f5410 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1338013380 con 0x7f13440f5410 2024-02-20T00:35:37.905 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1338014e50 con 0x7f13440f5410 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.898+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1344109bd0 con 0x7f13440f5410 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.899+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13380095f0 con 0x7f13440f5410 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.899+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344045ad0 con 0x7f13440f5410 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.899+0000 7f13417fa700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 0x7f13340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.899+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1338040e90 con 0x7f13440f5410 2024-02-20T00:35:37.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.899+0000 7f1343fff700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 0x7f13340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:37.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.900+0000 7f1343fff700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 0x7f13340323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f13300096a0 tx=0x7f1330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:37.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:37.904+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1338014420 con 0x7f13440f5410 2024-02-20T00:35:38.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.106+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13440fafa0 con 0x7f13440f5410 2024-02-20T00:35:38.108 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.107+0000 7f13417fa700 1 -- 172.21.15.195:0/1807756612 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1338010230 con 0x7f13440f5410 2024-02-20T00:35:38.108 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:38.109 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 msgr2=0x7f13340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 0x7f13340323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f13300096a0 tx=0x7f1330006b40).stop 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 msgr2=0x7f13441094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f1338005010 tx=0x7f133800b040).stop 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 shutdown_connections 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f133402ff10 0x7f13340323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 --2- 172.21.15.195:0/1807756612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13440f5410 0x7f13441094a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:38.110 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.108+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 >> 172.21.15.195:0/1807756612 conn(0x7f13440f0b70 msgr2=0x7f13440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:38.111 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.109+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 shutdown_connections 2024-02-20T00:35:38.111 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:38.109+0000 7f134ac8b700 1 -- 172.21.15.195:0/1807756612 wait complete. 2024-02-20T00:35:38.111 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:36.617127+0000 mgr.smithi087.iydsqo (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:38.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:38.107953+0000 mon.smithi087 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.195:0/1807756612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:39.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:39.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:40.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:38.617486+0000 mgr.smithi087.iydsqo (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:41.517 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 -- 172.21.15.195:0/1458940085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 msgr2=0x7f070c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:41.517 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 --2- 172.21.15.195:0/1458940085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c0f6930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f06fc009fb0 tx=0x7f06fc009b30).stop 2024-02-20T00:35:41.517 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 -- 172.21.15.195:0/1458940085 shutdown_connections 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 --2- 172.21.15.195:0/1458940085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c0f6930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 -- 172.21.15.195:0/1458940085 >> 172.21.15.195:0/1458940085 conn(0x7f070c0f1c70 msgr2=0x7f070c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 -- 172.21.15.195:0/1458940085 shutdown_connections 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.514+0000 7f071246f700 1 -- 172.21.15.195:0/1458940085 wait complete. 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f071246f700 1 Processor -- start 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f071246f700 1 -- start start 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f071246f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f071246f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c106cd0 con 0x7f070c0f6510 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f070bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:41.518 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f070bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60622/0 (socket says 172.21.15.195:60622) 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f070bfff700 1 -- 172.21.15.195:0/931193890 learned_addr learned my addr 172.21.15.195:0/931193890 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.515+0000 7f070bfff700 1 -- 172.21.15.195:0/931193890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc005040 con 0x7f070c0f6510 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f070bfff700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f06fc005010 tx=0x7f06fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc009320 con 0x7f070c0f6510 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c106ed0 con 0x7f070c0f6510 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06fc013380 con 0x7f070c0f6510 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06fc014e50 con 0x7f070c0f6510 2024-02-20T00:35:41.519 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.516+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c107360 con 0x7f070c0f6510 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06fc0095f0 con 0x7f070c0f6510 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070c045ad0 con 0x7f070c0f6510 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f0708ff9700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06fc040fa0 con 0x7f070c0f6510 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f070b7fe700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:41.520 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.517+0000 7f070b7fe700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 0x7f06f80323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f06f40096a0 tx=0x7f06f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:41.523 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.522+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06fc014420 con 0x7f070c0f6510 2024-02-20T00:35:41.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.727+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f070c108210 con 0x7f070c0f6510 2024-02-20T00:35:41.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.727+0000 7f0708ff9700 1 -- 172.21.15.195:0/931193890 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06fc010230 con 0x7f070c0f6510 2024-02-20T00:35:41.730 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:41.730 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 msgr2=0x7f06f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 0x7f06f80323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f06f40096a0 tx=0x7f06f4006b40).stop 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 msgr2=0x7f070c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f06fc005010 tx=0x7f06fc00b040).stop 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 shutdown_connections 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 --2- 172.21.15.195:0/931193890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f070c0f6510 0x7f070c1067e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 >> 172.21.15.195:0/931193890 conn(0x7f070c0f1c70 msgr2=0x7f070c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 shutdown_connections 2024-02-20T00:35:41.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:41.729+0000 7f071246f700 1 -- 172.21.15.195:0/931193890 wait complete. 2024-02-20T00:35:41.732 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:42.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:40.617797+0000 mgr.smithi087.iydsqo (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:42.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:41.728871+0000 mon.smithi087 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.195:0/931193890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:43.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:43.477 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:44.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:42.618072+0000 mgr.smithi087.iydsqo (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:45.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 -- 172.21.15.195:0/324652571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 msgr2=0x7fd6ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:45.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/324652571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec0f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fd6e8009fb0 tx=0x7fd6e8009b30).stop 2024-02-20T00:35:45.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 -- 172.21.15.195:0/324652571 shutdown_connections 2024-02-20T00:35:45.279 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/324652571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec0f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 -- 172.21.15.195:0/324652571 >> 172.21.15.195:0/324652571 conn(0x7fd6ec0f0b70 msgr2=0x7fd6ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.276+0000 7fd6f3c93700 1 -- 172.21.15.195:0/324652571 shutdown_connections 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f3c93700 1 -- 172.21.15.195:0/324652571 wait complete. 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f3c93700 1 Processor -- start 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f3c93700 1 -- start start 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f3c93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f3c93700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec060280 con 0x7fd6ec0f5410 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f1a2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f1a2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60638/0 (socket says 172.21.15.195:60638) 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.277+0000 7fd6f1a2f700 1 -- 172.21.15.195:0/2052667270 learned_addr learned my addr 172.21.15.195:0/2052667270 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6f1a2f700 1 -- 172.21.15.195:0/2052667270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e8005040 con 0x7fd6ec0f5410 2024-02-20T00:35:45.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6f1a2f700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd6e8005010 tx=0x7fd6e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e8009320 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec060480 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec109bd0 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6e8013380 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.278+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e8014df0 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.279+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d4004fa0 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.279+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6e8014420 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.279+0000 7fd6e27fc700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 0x7fd6d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.279+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6e8041070 con 0x7fd6ec0f5410 2024-02-20T00:35:45.282 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.279+0000 7fd6f122e700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 0x7fd6d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:45.283 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.280+0000 7fd6f122e700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 0x7fd6d80323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd6dc0096a0 tx=0x7fd6dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:45.285 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.284+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6e800fb30 con 0x7fd6ec0f5410 2024-02-20T00:35:45.489 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.488+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6d4005670 con 0x7fd6ec0f5410 2024-02-20T00:35:45.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.488+0000 7fd6e27fc700 1 -- 172.21.15.195:0/2052667270 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd6e800fb30 con 0x7fd6ec0f5410 2024-02-20T00:35:45.490 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:45.490 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:45.491 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 msgr2=0x7fd6d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 0x7fd6d80323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd6dc0096a0 tx=0x7fd6dc006b40).stop 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 msgr2=0x7fd6ec1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd6e8005010 tx=0x7fd6e800b040).stop 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 shutdown_connections 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd6d802ff10 0x7fd6d80323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 --2- 172.21.15.195:0/2052667270 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6ec0f5410 0x7fd6ec1094a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 >> 172.21.15.195:0/2052667270 conn(0x7fd6ec0f0b70 msgr2=0x7fd6ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 shutdown_connections 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:45.490+0000 7fd6f3c93700 1 -- 172.21.15.195:0/2052667270 wait complete. 2024-02-20T00:35:45.492 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:46.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:44.618421+0000 mgr.smithi087.iydsqo (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:45.489566+0000 mon.smithi087 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.195:0/2052667270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:47.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:47.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:46.618737+0000 mgr.smithi087.iydsqo (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- 172.21.15.195:0/2480131022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 msgr2=0x7f48980f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 --2- 172.21.15.195:0/2480131022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48980f8fa0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f488c009fb0 tx=0x7f488c009b30).stop 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- 172.21.15.195:0/2480131022 shutdown_connections 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 --2- 172.21.15.195:0/2480131022 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48980f8fa0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- 172.21.15.195:0/2480131022 >> 172.21.15.195:0/2480131022 conn(0x7f48980f09a0 msgr2=0x7f48980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:49.176 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- 172.21.15.195:0/2480131022 shutdown_connections 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- 172.21.15.195:0/2480131022 wait complete. 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 Processor -- start 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.173+0000 7f489d8dd700 1 -- start start 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489d8dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489d8dd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4898106ac0 con 0x7f4898060160 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489c8db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489c8db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35464/0 (socket says 172.21.15.195:35464) 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489c8db700 1 -- 172.21.15.195:0/1926170810 learned_addr learned my addr 172.21.15.195:0/1926170810 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489c8db700 1 -- 172.21.15.195:0/1926170810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c005040 con 0x7f4898060160 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489c8db700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f488c005010 tx=0x7f488c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c009320 con 0x7f4898060160 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4898106cc0 con 0x7f4898060160 2024-02-20T00:35:49.177 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f488c013380 con 0x7f4898060160 2024-02-20T00:35:49.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c014e50 con 0x7f4898060160 2024-02-20T00:35:49.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.174+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4898107150 con 0x7f4898060160 2024-02-20T00:35:49.178 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.175+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f488c0263a0 con 0x7f4898060160 2024-02-20T00:35:49.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.175+0000 7f48957fa700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 0x7f48880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:49.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.175+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4898045ad0 con 0x7f4898060160 2024-02-20T00:35:49.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.175+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f488c041270 con 0x7f4898060160 2024-02-20T00:35:49.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.176+0000 7f4897fff700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 0x7f48880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:49.179 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.176+0000 7f4897fff700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 0x7f48880323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f48840096a0 tx=0x7f4884006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:49.181 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.180+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f488c012920 con 0x7f4898060160 2024-02-20T00:35:49.388 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.386+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4898108000 con 0x7f4898060160 2024-02-20T00:35:49.389 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.387+0000 7f48957fa700 1 -- 172.21.15.195:0/1926170810 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f488c012330 con 0x7f4898060160 2024-02-20T00:35:49.389 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:49.389 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:49.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.388+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 msgr2=0x7f48880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:49.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 0x7f48880323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f48840096a0 tx=0x7f4884006b40).stop 2024-02-20T00:35:49.390 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 msgr2=0x7f48981065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f488c005010 tx=0x7f488c00b040).stop 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 shutdown_connections 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f488802ff10 0x7f48880323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 --2- 172.21.15.195:0/1926170810 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4898060160 0x7f48981065d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 >> 172.21.15.195:0/1926170810 conn(0x7f48980f09a0 msgr2=0x7f48980f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 shutdown_connections 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:49.389+0000 7f489d8dd700 1 -- 172.21.15.195:0/1926170810 wait complete. 2024-02-20T00:35:49.391 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:50.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:48.619015+0000 mgr.smithi087.iydsqo (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:50.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:49.388339+0000 mon.smithi087 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.195:0/1926170810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:51.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:51.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:52.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:50.619355+0000 mgr.smithi087.iydsqo (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 -- 172.21.15.195:0/3177964753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 msgr2=0x7ff4d40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 --2- 172.21.15.195:0/3177964753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d40f66c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7ff4c4009fb0 tx=0x7ff4c4009b30).stop 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 -- 172.21.15.195:0/3177964753 shutdown_connections 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 --2- 172.21.15.195:0/3177964753 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d40f66c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 -- 172.21.15.195:0/3177964753 >> 172.21.15.195:0/3177964753 conn(0x7ff4d40f1aa0 msgr2=0x7ff4d40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 -- 172.21.15.195:0/3177964753 shutdown_connections 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 -- 172.21.15.195:0/3177964753 wait complete. 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.906+0000 7ff4d9669700 1 Processor -- start 2024-02-20T00:35:52.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d9669700 1 -- start start 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d9669700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d9669700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d4106ac0 con 0x7ff4d40f62a0 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:35470/0 (socket says 172.21.15.195:35470) 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d3fff700 1 -- 172.21.15.195:0/123583728 learned_addr learned my addr 172.21.15.195:0/123583728 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.907+0000 7ff4d3fff700 1 -- 172.21.15.195:0/123583728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4005040 con 0x7ff4d40f62a0 2024-02-20T00:35:52.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d3fff700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7ff4c4009830 tx=0x7ff4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:52.911 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c4009320 con 0x7ff4d40f62a0 2024-02-20T00:35:52.911 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d4106cc0 con 0x7ff4d40f62a0 2024-02-20T00:35:52.911 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4c4013380 con 0x7ff4d40f62a0 2024-02-20T00:35:52.911 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c4014420 con 0x7ff4d40f62a0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.908+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d4107150 con 0x7ff4d40f62a0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.909+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4c401e420 con 0x7ff4d40f62a0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.909+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4045ad0 con 0x7ff4d40f62a0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.909+0000 7ff4d0ff9700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 0x7ff4c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.909+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4c4040e30 con 0x7ff4d40f62a0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.909+0000 7ff4d37fe700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 0x7ff4c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:52.912 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.910+0000 7ff4d37fe700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 0x7ff4c0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff4bc0096a0 tx=0x7ff4bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:52.915 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:52.914+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4c40095f0 con 0x7ff4d40f62a0 2024-02-20T00:35:53.124 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.122+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4d40fbe30 con 0x7ff4d40f62a0 2024-02-20T00:35:53.125 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.123+0000 7ff4d0ff9700 1 -- 172.21.15.195:0/123583728 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff4c401cb10 con 0x7ff4d40f62a0 2024-02-20T00:35:53.125 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:53.125 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 msgr2=0x7ff4c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 0x7ff4c0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff4bc0096a0 tx=0x7ff4bc006b40).stop 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 msgr2=0x7ff4d41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7ff4c4009830 tx=0x7ff4c400b040).stop 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 shutdown_connections 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff4c002ff60 0x7ff4c0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 --2- 172.21.15.195:0/123583728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4d40f62a0 0x7ff4d41065d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 >> 172.21.15.195:0/123583728 conn(0x7ff4d40f1aa0 msgr2=0x7ff4d40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:53.127 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 shutdown_connections 2024-02-20T00:35:53.128 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:53.125+0000 7ff4d9669700 1 -- 172.21.15.195:0/123583728 wait complete. 2024-02-20T00:35:53.128 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:53.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:53.124456+0000 mon.smithi087 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.195:0/123583728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:54.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:52.619666+0000 mgr.smithi087.iydsqo (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:54.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:54.804 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 -- 172.21.15.195:0/2161831095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 msgr2=0x7eff800f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 --2- 172.21.15.195:0/2161831095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff800f4bd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7eff70009fb0 tx=0x7eff70009b30).stop 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 -- 172.21.15.195:0/2161831095 shutdown_connections 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 --2- 172.21.15.195:0/2161831095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff800f4bd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 -- 172.21.15.195:0/2161831095 >> 172.21.15.195:0/2161831095 conn(0x7eff800f02b0 msgr2=0x7eff800f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:56.665 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 -- 172.21.15.195:0/2161831095 shutdown_connections 2024-02-20T00:35:56.666 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.663+0000 7eff88036700 1 -- 172.21.15.195:0/2161831095 wait complete. 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff88036700 1 Processor -- start 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff88036700 1 -- start start 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff88036700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff88036700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff80106c80 con 0x7eff800f47b0 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff85dd2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff85dd2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:48478/0 (socket says 172.21.15.195:48478) 2024-02-20T00:35:56.667 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.664+0000 7eff85dd2700 1 -- 172.21.15.195:0/804093348 learned_addr learned my addr 172.21.15.195:0/804093348 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff85dd2700 1 -- 172.21.15.195:0/804093348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70005040 con 0x7eff800f47b0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff85dd2700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7eff70005010 tx=0x7eff7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff70009320 con 0x7eff800f47b0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff80106e80 con 0x7eff800f47b0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff70013380 con 0x7eff800f47b0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff70014e50 con 0x7eff800f47b0 2024-02-20T00:35:56.668 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.665+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff80107310 con 0x7eff800f47b0 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.666+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff80045ad0 con 0x7eff800f47b0 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.666+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff700095f0 con 0x7eff800f47b0 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.666+0000 7eff767fc700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 0x7eff6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.666+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff70040e90 con 0x7eff800f47b0 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.666+0000 7eff855d1700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 0x7eff6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:35:56.669 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.667+0000 7eff855d1700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 0x7eff6c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7eff7c0096a0 tx=0x7eff7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:35:56.672 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.671+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff70014420 con 0x7eff800f47b0 2024-02-20T00:35:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:54.619946+0000 mgr.smithi087.iydsqo (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:56.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.878+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff801081c0 con 0x7eff800f47b0 2024-02-20T00:35:56.880 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.879+0000 7eff767fc700 1 -- 172.21.15.195:0/804093348 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff70010230 con 0x7eff800f47b0 2024-02-20T00:35:56.881 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:35:56.881 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:35:56.882 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 msgr2=0x7eff6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 0x7eff6c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7eff7c0096a0 tx=0x7eff7c006b40).stop 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 msgr2=0x7eff80106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7eff70005010 tx=0x7eff7000b040).stop 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 shutdown_connections 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7eff6c02ff10 0x7eff6c0323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 --2- 172.21.15.195:0/804093348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff800f47b0 0x7eff80106790 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 >> 172.21.15.195:0/804093348 conn(0x7eff800f02b0 msgr2=0x7eff800f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 shutdown_connections 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:35:56.881+0000 7eff88036700 1 -- 172.21.15.195:0/804093348 wait complete. 2024-02-20T00:35:56.883 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:35:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:56.674422+0000 mon.smithi087 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:35:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:56.674619+0000 mon.smithi087 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:35:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:56.880199+0000 mon.smithi087 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.195:0/804093348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:35:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:57.336282+0000 mon.smithi087 (mon.0) 298 : audit [DBG] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:35:58.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:35:58.517 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:35:58.853 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:56.620290+0000 mgr.smithi087.iydsqo (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:35:58.853 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:35:57.335691+0000 mgr.smithi087.iydsqo (mgr.14162) 309 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi087 ... 2024-02-20T00:35:58.853 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T00:35:58.853 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:34:04Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:34:42Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:35:19Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:35:58.854 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('uid/gid not found') 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: uid/gid not found 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: yield (conn, connr) 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: code, '\n'.join(err))) 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi087 ... 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi087 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:34:04Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:34:42Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.855 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr time="2024-02-20T00:35:19Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: Traceback (most recent call last): 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: main() 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: r = ctx.func(ctx) 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: return func(ctx) 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T00:35:58.856 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: File "/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:35:58.857 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: raise RuntimeError('uid/gid not found') 2024-02-20T00:35:58.857 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: RuntimeError: uid/gid not found 2024-02-20T00:35:58.857 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:57.562329+0000 mon.smithi087 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:35:58.857 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:57.570896+0000 mon.smithi087 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:35:58.857 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:35:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:57.808681+0000 mon.smithi087 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:36:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:35:58.620547+0000 mgr.smithi087.iydsqo (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:00.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:35:59.418043+0000 mon.smithi087 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:36:01.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:01 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:01.374663+0000 mon.smithi087 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.195:0/1466541324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 -- 172.21.15.195:0/1623177195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 msgr2=0x7fc4e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1623177195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e00f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fc4d4009fb0 tx=0x7fc4d4009b30).stop 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 -- 172.21.15.195:0/1623177195 shutdown_connections 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1623177195 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e00f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 -- 172.21.15.195:0/1623177195 >> 172.21.15.195:0/1623177195 conn(0x7fc4e00f0b70 msgr2=0x7fc4e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 -- 172.21.15.195:0/1623177195 shutdown_connections 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.156+0000 7fc4e5749700 1 -- 172.21.15.195:0/1623177195 wait complete. 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4e5749700 1 Processor -- start 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4e5749700 1 -- start start 2024-02-20T00:36:01.894 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4e5749700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4e5749700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e0106d10 con 0x7fc4e00f5410 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4deffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4deffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:48492/0 (socket says 172.21.15.195:48492) 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.157+0000 7fc4deffd700 1 -- 172.21.15.195:0/1466541324 learned_addr learned my addr 172.21.15.195:0/1466541324 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:01.895 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4deffd700 1 -- 172.21.15.195:0/1466541324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4005040 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4deffd700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc4d4005010 tx=0x7fc4d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d4009320 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e0106f10 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4d4013380 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d4014e50 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.158+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e01073a0 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4d40095f0 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0045ad0 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4c77fe700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 0x7fc4c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4d4040fa0 con 0x7fc4e00f5410 2024-02-20T00:36:01.896 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4de7fc700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 0x7fc4c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.159+0000 7fc4de7fc700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 0x7fc4c80323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc4d00096a0 tx=0x7fc4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.164+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4d4014420 con 0x7fc4e00f5410 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.373+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4e0108250 con 0x7fc4e00f5410 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.373+0000 7fc4c77fe700 1 -- 172.21.15.195:0/1466541324 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4d4010230 con 0x7fc4e00f5410 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 msgr2=0x7fc4c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 0x7fc4c80323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc4d00096a0 tx=0x7fc4d0006b40).stop 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 msgr2=0x7fc4e0106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc4d4005010 tx=0x7fc4d400b040).stop 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 shutdown_connections 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc4c802ff10 0x7fc4c80323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 --2- 172.21.15.195:0/1466541324 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4e00f5410 0x7fc4e0106820 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 >> 172.21.15.195:0/1466541324 conn(0x7fc4e00f0b70 msgr2=0x7fc4e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:01.897 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 shutdown_connections 2024-02-20T00:36:01.898 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:01.375+0000 7fc4e5749700 1 -- 172.21.15.195:0/1466541324 wait complete. 2024-02-20T00:36:01.898 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:02.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:00.620894+0000 mgr.smithi087.iydsqo (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:02.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:02.020472+0000 mon.smithi087 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:36:03.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:03.447 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:02.621207+0000 mgr.smithi087.iydsqo (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:03.237224+0000 mon.smithi087 (mon.0) 305 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:03.237847+0000 mgr.smithi087.iydsqo (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:03.240003+0000 mon.smithi087 (mon.0) 306 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cephadm 2024-02-20T00:36:03.242101+0000 mgr.smithi087.iydsqo (mgr.14162) 314 : cephadm [INF] Deploying daemon alertmanager.smithi087 on smithi087 2024-02-20T00:36:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:03.421856+0000 mon.smithi087 (mon.0) 307 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-20T00:36:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- 172.21.15.195:0/502975661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 msgr2=0x7f35bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/502975661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc0f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f35b0009fb0 tx=0x7f35b0009b30).stop 2024-02-20T00:36:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- 172.21.15.195:0/502975661 shutdown_connections 2024-02-20T00:36:05.325 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/502975661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc0f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- 172.21.15.195:0/502975661 >> 172.21.15.195:0/502975661 conn(0x7f35bc0f0b70 msgr2=0x7f35bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- 172.21.15.195:0/502975661 shutdown_connections 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- 172.21.15.195:0/502975661 wait complete. 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 Processor -- start 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.322+0000 7f35c2ce2700 1 -- start start 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c2ce2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c2ce2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc106ca0 con 0x7f35bc0f5410 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c0a7e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c0a7e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:48516/0 (socket says 172.21.15.195:48516) 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c0a7e700 1 -- 172.21.15.195:0/2576822162 learned_addr learned my addr 172.21.15.195:0/2576822162 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c0a7e700 1 -- 172.21.15.195:0/2576822162 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0005040 con 0x7f35bc0f5410 2024-02-20T00:36:05.326 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35c0a7e700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f35b00089f0 tx=0x7f35b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.323+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b0009320 con 0x7f35bc0f5410 2024-02-20T00:36:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35b0013380 con 0x7f35bc0f5410 2024-02-20T00:36:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc106ea0 con 0x7f35bc0f5410 2024-02-20T00:36:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35b001ed80 con 0x7f35bc0f5410 2024-02-20T00:36:05.327 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc107330 con 0x7f35bc0f5410 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35b001e420 con 0x7f35bc0f5410 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc045ad0 con 0x7f35bc0f5410 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35b97fa700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 0x7f35ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.324+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35b00410d0 con 0x7f35bc0f5410 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.325+0000 7f35bbfff700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 0x7f35ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:05.328 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.325+0000 7f35bbfff700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 0x7f35ac032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f35a80096a0 tx=0x7f35a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:05.331 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.329+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35b00095f0 con 0x7f35bc0f5410 2024-02-20T00:36:05.535 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.534+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35bc0fafa0 con 0x7f35bc0f5410 2024-02-20T00:36:05.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.534+0000 7f35b97fa700 1 -- 172.21.15.195:0/2576822162 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35b002c2d0 con 0x7f35bc0f5410 2024-02-20T00:36:05.536 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:05.536 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:05.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 msgr2=0x7f35ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 0x7f35ac032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f35a80096a0 tx=0x7f35a8006b40).stop 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 msgr2=0x7f35bc1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f35b00089f0 tx=0x7f35b000b040).stop 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 shutdown_connections 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f35ac02ff60 0x7f35ac032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 --2- 172.21.15.195:0/2576822162 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35bc0f5410 0x7f35bc1067b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 >> 172.21.15.195:0/2576822162 conn(0x7f35bc0f0b70 msgr2=0x7f35bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 shutdown_connections 2024-02-20T00:36:05.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:05.536+0000 7f35c2ce2700 1 -- 172.21.15.195:0/2576822162 wait complete. 2024-02-20T00:36:05.539 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:06.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:05.238145+0000 mgr.smithi087.iydsqo (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:06.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:05.535624+0000 mon.smithi087 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.195:0/2576822162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:07.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:07.329 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:08.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:07.238498+0000 mgr.smithi087.iydsqo (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:09.158 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 -- 172.21.15.195:0/725347408 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 msgr2=0x7f65b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 --2- 172.21.15.195:0/725347408 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b40f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f65a8009fb0 tx=0x7f65a8009b30).stop 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 -- 172.21.15.195:0/725347408 shutdown_connections 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 --2- 172.21.15.195:0/725347408 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b40f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 -- 172.21.15.195:0/725347408 >> 172.21.15.195:0/725347408 conn(0x7f65b40f09a0 msgr2=0x7f65b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 -- 172.21.15.195:0/725347408 shutdown_connections 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.156+0000 7f65b8809700 1 -- 172.21.15.195:0/725347408 wait complete. 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b8809700 1 Processor -- start 2024-02-20T00:36:09.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b8809700 1 -- start start 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b8809700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b8809700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4105990 con 0x7f65b40f51a0 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:33418/0 (socket says 172.21.15.195:33418) 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.157+0000 7f65b37fe700 1 -- 172.21.15.195:0/2892011314 learned_addr learned my addr 172.21.15.195:0/2892011314 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:09.163 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f65b37fe700 1 -- 172.21.15.195:0/2892011314 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a8005040 con 0x7f65b40f51a0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f65b37fe700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f65a8009830 tx=0x7f65a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65a8009320 con 0x7f65b40f51a0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65a8013380 con 0x7f65b40f51a0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65a8014420 con 0x7f65b40f51a0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b4105b90 con 0x7f65b40f51a0 2024-02-20T00:36:09.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.158+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4106020 con 0x7f65b40f51a0 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65a801e420 con 0x7f65b40f51a0 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4045ad0 con 0x7f65b40f51a0 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f659bfff700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65a8040e30 con 0x7f65b40f51a0 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f65b2ffd700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:09.165 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.159+0000 7f65b2ffd700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 0x7f659c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f65a40096a0 tx=0x7f65a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:09.166 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.164+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65a801cb10 con 0x7f65b40f51a0 2024-02-20T00:36:09.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.368+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65b40fad30 con 0x7f65b40f51a0 2024-02-20T00:36:09.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.369+0000 7f659bfff700 1 -- 172.21.15.195:0/2892011314 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65a8004490 con 0x7f65b40f51a0 2024-02-20T00:36:09.371 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:09.371 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:09.372 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 msgr2=0x7f659c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 0x7f659c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f65a40096a0 tx=0x7f65a4006b40).stop 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 msgr2=0x7f65b41054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f65a8009830 tx=0x7f65a800b040).stop 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 shutdown_connections 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 --2- 172.21.15.195:0/2892011314 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f65b40f51a0 0x7f65b41054a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 >> 172.21.15.195:0/2892011314 conn(0x7f65b40f09a0 msgr2=0x7f65b40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 shutdown_connections 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:09.371+0000 7f65b8809700 1 -- 172.21.15.195:0/2892011314 wait complete. 2024-02-20T00:36:09.373 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:10.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:09.238895+0000 mgr.smithi087.iydsqo (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:10.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:09.370424+0000 mon.smithi087 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.195:0/2892011314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:11.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:11.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:12.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:11.239241+0000 mgr.smithi087.iydsqo (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:13.054 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 -- 172.21.15.195:0/901871305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780f5410 msgr2=0x7f1b780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 --2- 172.21.15.195:0/901871305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780f5410 0x7f1b780f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f1b68009fb0 tx=0x7f1b68009b30).stop 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 -- 172.21.15.195:0/901871305 shutdown_connections 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 --2- 172.21.15.195:0/901871305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780f5410 0x7f1b780f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 -- 172.21.15.195:0/901871305 >> 172.21.15.195:0/901871305 conn(0x7f1b780f0b70 msgr2=0x7f1b780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 -- 172.21.15.195:0/901871305 shutdown_connections 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.052+0000 7f1b7e423700 1 -- 172.21.15.195:0/901871305 wait complete. 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b7e423700 1 Processor -- start 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b7e423700 1 -- start start 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b7e423700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:13.055 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b7e423700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b780fd940 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b77fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b77fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:33444/0 (socket says 172.21.15.195:33444) 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.053+0000 7f1b77fff700 1 -- 172.21.15.195:0/2657242075 learned_addr learned my addr 172.21.15.195:0/2657242075 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b77fff700 1 -- 172.21.15.195:0/2657242075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68005040 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b77fff700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1b68009f80 tx=0x7f1b68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b68013380 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b780fbed0 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b68013790 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b68014420 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.054+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b780fc3c0 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.055+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b68025060 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.055+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b78059100 con 0x7f1b780fd520 2024-02-20T00:36:13.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.055+0000 7f1b74ff9700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:13.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.055+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b68040d80 con 0x7f1b780fd520 2024-02-20T00:36:13.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.055+0000 7f1b777fe700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:13.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.056+0000 7f1b777fe700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 0x7f1b60032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1b6c0096a0 tx=0x7f1b6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:13.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.060+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b6800f280 con 0x7f1b780fd520 2024-02-20T00:36:13.269 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.267+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b78045ad0 con 0x7f1b780fd520 2024-02-20T00:36:13.270 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.268+0000 7f1b74ff9700 1 -- 172.21.15.195:0/2657242075 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b6802d2d0 con 0x7f1b780fd520 2024-02-20T00:36:13.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:13.270 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:13.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 msgr2=0x7f1b60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 0x7f1b60032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1b6c0096a0 tx=0x7f1b6c006b40).stop 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 msgr2=0x7f1b780fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1b68009f80 tx=0x7f1b68009b30).stop 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 shutdown_connections 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 --2- 172.21.15.195:0/2657242075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b780fd520 0x7f1b780fb9e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 >> 172.21.15.195:0/2657242075 conn(0x7f1b780f0b70 msgr2=0x7f1b7805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 shutdown_connections 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:13.270+0000 7f1b7e423700 1 -- 172.21.15.195:0/2657242075 wait complete. 2024-02-20T00:36:13.272 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:13.239619+0000 mgr.smithi087.iydsqo (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:14.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:13.269331+0000 mon.smithi087 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.195:0/2657242075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:15.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:15.035 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:16.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:15.239880+0000 mgr.smithi087.iydsqo (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 -- 172.21.15.195:0/3508281241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 msgr2=0x7f44400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 --2- 172.21.15.195:0/3508281241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44400f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f4434009fb0 tx=0x7f4434009b30).stop 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 -- 172.21.15.195:0/3508281241 shutdown_connections 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 --2- 172.21.15.195:0/3508281241 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44400f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 -- 172.21.15.195:0/3508281241 >> 172.21.15.195:0/3508281241 conn(0x7f44400f0b70 msgr2=0x7f44400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:16.932 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 -- 172.21.15.195:0/3508281241 shutdown_connections 2024-02-20T00:36:16.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.930+0000 7f4446c3f700 1 -- 172.21.15.195:0/3508281241 wait complete. 2024-02-20T00:36:16.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f4446c3f700 1 Processor -- start 2024-02-20T00:36:16.933 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f4446c3f700 1 -- start start 2024-02-20T00:36:16.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f4446c3f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:16.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f4446c3f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4440060280 con 0x7f44400f5410 2024-02-20T00:36:16.934 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f44449db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f44449db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:42174/0 (socket says 172.21.15.195:42174) 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f44449db700 1 -- 172.21.15.195:0/2935418720 learned_addr learned my addr 172.21.15.195:0/2935418720 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.931+0000 7f44449db700 1 -- 172.21.15.195:0/2935418720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434005040 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f44449db700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4434005010 tx=0x7f443400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4434009320 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4434013380 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4440060480 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4434014e50 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.932+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4440109bd0 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44340263a0 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f443d7fa700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 0x7f44300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4434041000 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440059100 con 0x7f44400f5410 2024-02-20T00:36:16.935 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f443ffff700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 0x7f44300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:16.936 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.933+0000 7f443ffff700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 0x7f44300323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f442c0096a0 tx=0x7f442c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:16.939 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:16.938+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4434012920 con 0x7f44400f5410 2024-02-20T00:36:17.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.141+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4440045ad0 con 0x7f44400f5410 2024-02-20T00:36:17.143 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.142+0000 7f443d7fa700 1 -- 172.21.15.195:0/2935418720 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f443402b020 con 0x7f44400f5410 2024-02-20T00:36:17.144 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:17.144 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 msgr2=0x7f44300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 0x7f44300323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f442c0096a0 tx=0x7f442c006b40).stop 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 msgr2=0x7f44401094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f4434005010 tx=0x7f443400b040).stop 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 shutdown_connections 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f443002ff10 0x7f44300323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 --2- 172.21.15.195:0/2935418720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f44400f5410 0x7f44401094a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:17.146 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 >> 172.21.15.195:0/2935418720 conn(0x7f44400f0b70 msgr2=0x7f44400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:17.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 shutdown_connections 2024-02-20T00:36:17.147 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:17.144+0000 7f4446c3f700 1 -- 172.21.15.195:0/2935418720 wait complete. 2024-02-20T00:36:17.147 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:17.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:17 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:17.143350+0000 mon.smithi087 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.195:0/2935418720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:18.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:17.240173+0000 mgr.smithi087.iydsqo (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:18.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:18.923 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:19.240538+0000 mgr.smithi087.iydsqo (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:20.854 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 -- 172.21.15.195:0/1071404984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 msgr2=0x7fcc4c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 --2- 172.21.15.195:0/1071404984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c0f66c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fcc48009fb0 tx=0x7fcc48009b30).stop 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 -- 172.21.15.195:0/1071404984 shutdown_connections 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 --2- 172.21.15.195:0/1071404984 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c0f66c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 -- 172.21.15.195:0/1071404984 >> 172.21.15.195:0/1071404984 conn(0x7fcc4c0f1aa0 msgr2=0x7fcc4c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 -- 172.21.15.195:0/1071404984 shutdown_connections 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.851+0000 7fcc533d4700 1 -- 172.21.15.195:0/1071404984 wait complete. 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc533d4700 1 Processor -- start 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc533d4700 1 -- start start 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc533d4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc533d4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c106ac0 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc523d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc523d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:42188/0 (socket says 172.21.15.195:42188) 2024-02-20T00:36:20.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc523d2700 1 -- 172.21.15.195:0/3346230310 learned_addr learned my addr 172.21.15.195:0/3346230310 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.852+0000 7fcc523d2700 1 -- 172.21.15.195:0/3346230310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48005040 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc523d2700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fcc480089f0 tx=0x7fcc4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc48009320 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c106cc0 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc48013380 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc48014e20 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.853+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c107150 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc480263a0 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c045ad0 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc42ffd700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 0x7fcc38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc480411f0 con 0x7fcc4c0f62a0 2024-02-20T00:36:20.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc51bd1700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 0x7fcc38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:20.857 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.854+0000 7fcc51bd1700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 0x7fcc38032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcc3c0096a0 tx=0x7fcc3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:20.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:20.859+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc4801cb10 con 0x7fcc4c0f62a0 2024-02-20T00:36:21.064 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.064+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc4c108000 con 0x7fcc4c0f62a0 2024-02-20T00:36:21.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.065+0000 7fcc42ffd700 1 -- 172.21.15.195:0/3346230310 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc4801cb10 con 0x7fcc4c0f62a0 2024-02-20T00:36:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:21.065 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:21.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 msgr2=0x7fcc38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:21.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 0x7fcc38032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcc3c0096a0 tx=0x7fcc3c006b40).stop 2024-02-20T00:36:21.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 msgr2=0x7fcc4c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:21.067 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fcc480089f0 tx=0x7fcc4800b040).stop 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 shutdown_connections 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fcc3802ff60 0x7fcc38032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 --2- 172.21.15.195:0/3346230310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcc4c0f62a0 0x7fcc4c1065d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 >> 172.21.15.195:0/3346230310 conn(0x7fcc4c0f1aa0 msgr2=0x7fcc4c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 shutdown_connections 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:21.066+0000 7fcc533d4700 1 -- 172.21.15.195:0/3346230310 wait complete. 2024-02-20T00:36:21.068 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:21.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:21 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:21.064888+0000 mon.smithi087 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.195:0/3346230310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:21.240918+0000 mgr.smithi087.iydsqo (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:22.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:22.856 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:24.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:23.241170+0000 mgr.smithi087.iydsqo (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 -- 172.21.15.195:0/2428371030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 msgr2=0x7f936c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 --2- 172.21.15.195:0/2428371030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c0f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f935c009fb0 tx=0x7f935c009b30).stop 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 -- 172.21.15.195:0/2428371030 shutdown_connections 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 --2- 172.21.15.195:0/2428371030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c0f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 -- 172.21.15.195:0/2428371030 >> 172.21.15.195:0/2428371030 conn(0x7f936c0f0b70 msgr2=0x7f936c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 -- 172.21.15.195:0/2428371030 shutdown_connections 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.817+0000 7f93737b7700 1 -- 172.21.15.195:0/2428371030 wait complete. 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f93737b7700 1 Processor -- start 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f93737b7700 1 -- start start 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f93737b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:24.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f93737b7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c106d00 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f9371553700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f9371553700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:42208/0 (socket says 172.21.15.195:42208) 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.818+0000 7f9371553700 1 -- 172.21.15.195:0/569982388 learned_addr learned my addr 172.21.15.195:0/569982388 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f9371553700 1 -- 172.21.15.195:0/569982388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c005040 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f9371553700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f935c005010 tx=0x7f935c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c009320 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c106f00 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f935c013380 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c014e50 con 0x7f936c0f5410 2024-02-20T00:36:24.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.819+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c107390 con 0x7f936c0f5410 2024-02-20T00:36:24.822 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.820+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f935c0095f0 con 0x7f936c0f5410 2024-02-20T00:36:24.822 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.820+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c045ad0 con 0x7f936c0f5410 2024-02-20T00:36:24.822 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.820+0000 7f9361ffb700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 0x7f93580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:24.822 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.820+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f935c040e90 con 0x7f936c0f5410 2024-02-20T00:36:24.822 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.820+0000 7f9370d52700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 0x7f93580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:24.823 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.821+0000 7f9370d52700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 0x7f93580323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f93680096a0 tx=0x7f9368006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:24.825 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:24.825+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f935c014420 con 0x7f936c0f5410 2024-02-20T00:36:25.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.026+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f936c0fafa0 con 0x7f936c0f5410 2024-02-20T00:36:25.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.027+0000 7f9361ffb700 1 -- 172.21.15.195:0/569982388 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f935c010230 con 0x7f936c0f5410 2024-02-20T00:36:25.029 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:25.029 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:25.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 msgr2=0x7f93580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:25.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 0x7f93580323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f93680096a0 tx=0x7f9368006b40).stop 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 msgr2=0x7f936c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f935c005010 tx=0x7f935c00b040).stop 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 shutdown_connections 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f935802ff10 0x7f93580323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 --2- 172.21.15.195:0/569982388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f936c0f5410 0x7f936c106810 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 >> 172.21.15.195:0/569982388 conn(0x7f936c0f0b70 msgr2=0x7f936c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 shutdown_connections 2024-02-20T00:36:25.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:25.029+0000 7f93737b7700 1 -- 172.21.15.195:0/569982388 wait complete. 2024-02-20T00:36:25.032 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:25.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:25 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:25.028160+0000 mon.smithi087 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.195:0/569982388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:26.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:25.241524+0000 mgr.smithi087.iydsqo (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:26.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:26.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.641+0000 7ff082252700 1 -- 172.21.15.195:0/2872995060 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 msgr2=0x7ff07c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.641+0000 7ff082252700 1 --2- 172.21.15.195:0/2872995060 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c0f90f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7ff06c009fb0 tx=0x7ff06c009b30).stop 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.641+0000 7ff082252700 1 -- 172.21.15.195:0/2872995060 shutdown_connections 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.641+0000 7ff082252700 1 --2- 172.21.15.195:0/2872995060 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c0f90f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.641+0000 7ff082252700 1 -- 172.21.15.195:0/2872995060 >> 172.21.15.195:0/2872995060 conn(0x7ff07c0f0b70 msgr2=0x7ff07c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 -- 172.21.15.195:0/2872995060 shutdown_connections 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 -- 172.21.15.195:0/2872995060 wait complete. 2024-02-20T00:36:28.644 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 Processor -- start 2024-02-20T00:36:28.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 -- start start 2024-02-20T00:36:28.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:28.645 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.642+0000 7ff082252700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c106c70 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60686/0 (socket says 172.21.15.195:60686) 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff07b7fe700 1 -- 172.21.15.195:0/4147212357 learned_addr learned my addr 172.21.15.195:0/4147212357 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff07b7fe700 1 -- 172.21.15.195:0/4147212357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c005040 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff07b7fe700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7ff06c0089f0 tx=0x7ff06c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c009320 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c106e70 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff06c013380 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c014e20 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.643+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c107300 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.644+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff06c014420 con 0x7ff07c060280 2024-02-20T00:36:28.647 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.644+0000 7ff063fff700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:28.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.644+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff06c040ee0 con 0x7ff07c060280 2024-02-20T00:36:28.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.644+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c045ad0 con 0x7ff07c060280 2024-02-20T00:36:28.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.645+0000 7ff07affd700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:28.648 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.645+0000 7ff07affd700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 0x7ff064032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff0700096a0 tx=0x7ff070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:28.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.649+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff06c014790 con 0x7ff07c060280 2024-02-20T00:36:28.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:27.241827+0000 mgr.smithi087.iydsqo (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:28.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.855+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff07c108020 con 0x7ff07c060280 2024-02-20T00:36:28.857 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.856+0000 7ff063fff700 1 -- 172.21.15.195:0/4147212357 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff06c010230 con 0x7ff07c060280 2024-02-20T00:36:28.858 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:28.858 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:28.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 msgr2=0x7ff064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:28.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 0x7ff064032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ff0700096a0 tx=0x7ff070006b40).stop 2024-02-20T00:36:28.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 msgr2=0x7ff07c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7ff06c0089f0 tx=0x7ff06c00b040).stop 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 shutdown_connections 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff06402ff60 0x7ff064032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 --2- 172.21.15.195:0/4147212357 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff07c060280 0x7ff07c106780 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 >> 172.21.15.195:0/4147212357 conn(0x7ff07c0f0b70 msgr2=0x7ff07c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:28.860 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 shutdown_connections 2024-02-20T00:36:28.861 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:28.857+0000 7ff082252700 1 -- 172.21.15.195:0/4147212357 wait complete. 2024-02-20T00:36:28.861 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:29.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:29 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:28.856713+0000 mon.smithi087 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.195:0/4147212357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:30.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:30.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:29.242173+0000 mgr.smithi087.iydsqo (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:32.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.463+0000 7fa0b89ec700 1 -- 172.21.15.195:0/82411161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 msgr2=0x7fa0b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:32.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.463+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/82411161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b40f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fa0a4009fb0 tx=0x7fa0a4009b30).stop 2024-02-20T00:36:32.470 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.463+0000 7fa0b89ec700 1 -- 172.21.15.195:0/82411161 shutdown_connections 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.463+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/82411161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b40f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.463+0000 7fa0b89ec700 1 -- 172.21.15.195:0/82411161 >> 172.21.15.195:0/82411161 conn(0x7fa0b40f09a0 msgr2=0x7fa0b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 -- 172.21.15.195:0/82411161 shutdown_connections 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 -- 172.21.15.195:0/82411161 wait complete. 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 Processor -- start 2024-02-20T00:36:32.471 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 -- start start 2024-02-20T00:36:32.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:32.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.464+0000 7fa0b89ec700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4060280 con 0x7fa0b40f51a0 2024-02-20T00:36:32.472 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:32.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60700/0 (socket says 172.21.15.195:60700) 2024-02-20T00:36:32.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b37fe700 1 -- 172.21.15.195:0/2275095419 learned_addr learned my addr 172.21.15.195:0/2275095419 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:32.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b37fe700 1 -- 172.21.15.195:0/2275095419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4005040 con 0x7fa0b40f51a0 2024-02-20T00:36:32.473 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b37fe700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa0a40089f0 tx=0x7fa0a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:32.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a4009320 con 0x7fa0b40f51a0 2024-02-20T00:36:32.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b4060480 con 0x7fa0b40f51a0 2024-02-20T00:36:32.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0a4013380 con 0x7fa0b40f51a0 2024-02-20T00:36:32.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.465+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a4014e20 con 0x7fa0b40f51a0 2024-02-20T00:36:32.474 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.466+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41077b0 con 0x7fa0b40f51a0 2024-02-20T00:36:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.466+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0a4014420 con 0x7fa0b40f51a0 2024-02-20T00:36:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.466+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b4045ad0 con 0x7fa0b40f51a0 2024-02-20T00:36:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.466+0000 7fa09bfff700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.466+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0a4040fc0 con 0x7fa0b40f51a0 2024-02-20T00:36:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.467+0000 7fa0b2ffd700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:32.476 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.467+0000 7fa0b2ffd700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 0x7fa09c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa0a80096a0 tx=0x7fa0a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:32.476 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.471+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0a401db70 con 0x7fa0b40f51a0 2024-02-20T00:36:32.678 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.676+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0b40fad30 con 0x7fa0b40f51a0 2024-02-20T00:36:32.679 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.677+0000 7fa09bfff700 1 -- 172.21.15.195:0/2275095419 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0a401db70 con 0x7fa0b40f51a0 2024-02-20T00:36:32.679 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:32.679 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 msgr2=0x7fa09c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 0x7fa09c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa0a80096a0 tx=0x7fa0a8006b40).stop 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 msgr2=0x7fa0b4107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa0a40089f0 tx=0x7fa0a400b040).stop 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 shutdown_connections 2024-02-20T00:36:32.681 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fa09c02ff60 0x7fa09c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 --2- 172.21.15.195:0/2275095419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0b40f51a0 0x7fa0b4107080 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 >> 172.21.15.195:0/2275095419 conn(0x7fa0b40f09a0 msgr2=0x7fa0b40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 shutdown_connections 2024-02-20T00:36:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:32.679+0000 7fa0b89ec700 1 -- 172.21.15.195:0/2275095419 wait complete. 2024-02-20T00:36:32.682 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:32.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:31.242456+0000 mgr.smithi087.iydsqo (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:33.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:33 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:32.678334+0000 mon.smithi087 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.195:0/2275095419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:34.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:34.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:34.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:33.242739+0000 mgr.smithi087.iydsqo (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:36.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 -- 172.21.15.195:0/2159209571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 msgr2=0x7f86180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:36.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 --2- 172.21.15.195:0/2159209571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f86180f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f8600009fb0 tx=0x7f8600009b30).stop 2024-02-20T00:36:36.273 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 -- 172.21.15.195:0/2159209571 shutdown_connections 2024-02-20T00:36:36.273 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 --2- 172.21.15.195:0/2159209571 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f86180f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:36.273 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 -- 172.21.15.195:0/2159209571 >> 172.21.15.195:0/2159209571 conn(0x7f86180f0b70 msgr2=0x7f86180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:36.273 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 -- 172.21.15.195:0/2159209571 shutdown_connections 2024-02-20T00:36:36.273 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.270+0000 7f861d2f9700 1 -- 172.21.15.195:0/2159209571 wait complete. 2024-02-20T00:36:36.274 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.271+0000 7f861d2f9700 1 Processor -- start 2024-02-20T00:36:36.274 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.271+0000 7f861d2f9700 1 -- start start 2024-02-20T00:36:36.274 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f861d2f9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:36.274 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f861d2f9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8618105c10 con 0x7f86180f5410 2024-02-20T00:36:36.274 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f8616ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f8616ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60716/0 (socket says 172.21.15.195:60716) 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f8616ffd700 1 -- 172.21.15.195:0/3301290845 learned_addr learned my addr 172.21.15.195:0/3301290845 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f8616ffd700 1 -- 172.21.15.195:0/3301290845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8600005040 con 0x7f86180f5410 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f8616ffd700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f86000089f0 tx=0x7f860000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8600009320 con 0x7f86180f5410 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8618105e10 con 0x7f86180f5410 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8600013380 con 0x7f86180f5410 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.272+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f860001ed80 con 0x7f86180f5410 2024-02-20T00:36:36.275 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.273+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86181062a0 con 0x7f86180f5410 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.273+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f860001e420 con 0x7f86180f5410 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.273+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8618045ad0 con 0x7f86180f5410 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.273+0000 7f860f7fe700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.273+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8600041050 con 0x7f86180f5410 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.274+0000 7f86167fc700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:36.276 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.274+0000 7f86167fc700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 0x7f8604032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f86080096a0 tx=0x7f8608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:36.280 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.278+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f860001cb10 con 0x7f86180f5410 2024-02-20T00:36:36.485 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.484+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86181070e0 con 0x7f86180f5410 2024-02-20T00:36:36.486 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.484+0000 7f860f7fe700 1 -- 172.21.15.195:0/3301290845 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8600004490 con 0x7f86180f5410 2024-02-20T00:36:36.487 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:36.487 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:36.489 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 msgr2=0x7f8604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:36.489 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 0x7f8604032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f86080096a0 tx=0x7f8608006b40).stop 2024-02-20T00:36:36.489 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 msgr2=0x7f8618105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:36.489 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f86000089f0 tx=0x7f860000b040).stop 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 shutdown_connections 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f860402ff60 0x7f8604032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 --2- 172.21.15.195:0/3301290845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86180f5410 0x7f8618105720 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 >> 172.21.15.195:0/3301290845 conn(0x7f86180f0b70 msgr2=0x7f86180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 shutdown_connections 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:36.486+0000 7f861d2f9700 1 -- 172.21.15.195:0/3301290845 wait complete. 2024-02-20T00:36:36.490 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:36.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:35.242995+0000 mgr.smithi087.iydsqo (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:37.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:37 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:36.485675+0000 mon.smithi087 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.195:0/3301290845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:38.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:38.281 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:37.243357+0000 mgr.smithi087.iydsqo (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:40.184 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 -- 172.21.15.195:0/2192196383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 msgr2=0x7f1f040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 --2- 172.21.15.195:0/2192196383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f040f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1ef8009fb0 tx=0x7f1ef8009b30).stop 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 -- 172.21.15.195:0/2192196383 shutdown_connections 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 --2- 172.21.15.195:0/2192196383 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f040f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 -- 172.21.15.195:0/2192196383 >> 172.21.15.195:0/2192196383 conn(0x7f1f040f1c70 msgr2=0x7f1f040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 -- 172.21.15.195:0/2192196383 shutdown_connections 2024-02-20T00:36:40.185 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.178+0000 7f1f0c617700 1 -- 172.21.15.195:0/2192196383 wait complete. 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0c617700 1 Processor -- start 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0c617700 1 -- start start 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0c617700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0c617700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04106cd0 con 0x7f1f040f6510 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0a3b3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0a3b3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46748/0 (socket says 172.21.15.195:46748) 2024-02-20T00:36:40.186 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.179+0000 7f1f0a3b3700 1 -- 172.21.15.195:0/1675661775 learned_addr learned my addr 172.21.15.195:0/1675661775 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1f0a3b3700 1 -- 172.21.15.195:0/1675661775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef8005040 con 0x7f1f040f6510 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1f0a3b3700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1ef8005010 tx=0x7f1ef800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ef8009320 con 0x7f1f040f6510 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f04106ed0 con 0x7f1f040f6510 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ef8013380 con 0x7f1f040f6510 2024-02-20T00:36:40.187 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ef8014e50 con 0x7f1f040f6510 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.180+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f04107360 con 0x7f1f040f6510 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.181+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ef80263a0 con 0x7f1f040f6510 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.181+0000 7f1ef6ffd700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 0x7f1ef00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.181+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04045ad0 con 0x7f1f040f6510 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.181+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ef8040fb0 con 0x7f1f040f6510 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.181+0000 7f1f09bb2700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 0x7f1ef00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:40.188 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.182+0000 7f1f09bb2700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 0x7f1ef00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:40.189 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.186+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ef8012920 con 0x7f1f040f6510 2024-02-20T00:36:40.391 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.390+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f041081b0 con 0x7f1f040f6510 2024-02-20T00:36:40.392 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.390+0000 7f1ef6ffd700 1 -- 172.21.15.195:0/1675661775 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ef802c090 con 0x7f1f040f6510 2024-02-20T00:36:40.392 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:40.392 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 msgr2=0x7f1ef00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 0x7f1ef00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f1f000096a0 tx=0x7f1f00006b40).stop 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 msgr2=0x7f1f041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1ef8005010 tx=0x7f1ef800b040).stop 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 shutdown_connections 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f1ef002ff10 0x7f1ef00323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 --2- 172.21.15.195:0/1675661775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f040f6510 0x7f1f041067e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 >> 172.21.15.195:0/1675661775 conn(0x7f1f040f1c70 msgr2=0x7f1f040f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 shutdown_connections 2024-02-20T00:36:40.394 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:40.392+0000 7f1f0c617700 1 -- 172.21.15.195:0/1675661775 wait complete. 2024-02-20T00:36:40.395 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:40.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:39.243760+0000 mgr.smithi087.iydsqo (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:41.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:41 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:40.391543+0000 mon.smithi087 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.195:0/1675661775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:42.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:42.101 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:42.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:41.244133+0000 mgr.smithi087.iydsqo (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 -- 172.21.15.195:0/1838738975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 msgr2=0x7f76f00f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 --2- 172.21.15.195:0/1838738975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f00f6ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f76e4009fb0 tx=0x7f76e4009b30).stop 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 -- 172.21.15.195:0/1838738975 shutdown_connections 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 --2- 172.21.15.195:0/1838738975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f00f6ea0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 -- 172.21.15.195:0/1838738975 >> 172.21.15.195:0/1838738975 conn(0x7f76f00f02e0 msgr2=0x7f76f00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 -- 172.21.15.195:0/1838738975 shutdown_connections 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 -- 172.21.15.195:0/1838738975 wait complete. 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.897+0000 7f76f6b37700 1 Processor -- start 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f6b37700 1 -- start start 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f6b37700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:43.900 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f6b37700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0106ca0 con 0x7f76f00f6a80 2024-02-20T00:36:43.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f48d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:43.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f48d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46764/0 (socket says 172.21.15.195:46764) 2024-02-20T00:36:43.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f48d3700 1 -- 172.21.15.195:0/2274111399 learned_addr learned my addr 172.21.15.195:0/2274111399 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:43.901 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f48d3700 1 -- 172.21.15.195:0/2274111399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4005040 con 0x7f76f00f6a80 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.898+0000 7f76f48d3700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f76e40089f0 tx=0x7f76e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.899+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e4009320 con 0x7f76f00f6a80 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.899+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76e4013380 con 0x7f76f00f6a80 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.899+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f0106ea0 con 0x7f76f00f6a80 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.899+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e4014e20 con 0x7f76f00f6a80 2024-02-20T00:36:43.902 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.899+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f0107330 con 0x7f76f00f6a80 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76e40095f0 con 0x7f76f00f6a80 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0045ad0 con 0x7f76f00f6a80 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76ed7fa700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 0x7f76e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76e4040e90 con 0x7f76f00f6a80 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76effff700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 0x7f76e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:43.903 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.900+0000 7f76effff700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 0x7f76e00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f76dc0096a0 tx=0x7f76dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:43.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:43.904+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76e4014420 con 0x7f76f00f6a80 2024-02-20T00:36:44.114 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.113+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76f01081e0 con 0x7f76f00f6a80 2024-02-20T00:36:44.115 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.113+0000 7f76ed7fa700 1 -- 172.21.15.195:0/2274111399 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76e401d980 con 0x7f76f00f6a80 2024-02-20T00:36:44.115 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:44.115 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:44.116 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 msgr2=0x7f76e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 0x7f76e00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f76dc0096a0 tx=0x7f76dc006b40).stop 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 msgr2=0x7f76f01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f76e40089f0 tx=0x7f76e400b040).stop 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 shutdown_connections 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f76e002ff10 0x7f76e00323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 --2- 172.21.15.195:0/2274111399 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f76f00f6a80 0x7f76f01067b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 >> 172.21.15.195:0/2274111399 conn(0x7f76f00f02e0 msgr2=0x7f76f00f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 shutdown_connections 2024-02-20T00:36:44.117 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:44.115+0000 7f76f6b37700 1 -- 172.21.15.195:0/2274111399 wait complete. 2024-02-20T00:36:44.118 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:43.244540+0000 mgr.smithi087.iydsqo (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:44.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:44.114705+0000 mon.smithi087 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.195:0/2274111399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:45.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:45.865 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:46.785 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:45.244909+0000 mgr.smithi087.iydsqo (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:47.786 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- 172.21.15.195:0/4197827333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 msgr2=0x7f81880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 --2- 172.21.15.195:0/4197827333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81880f66c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f817c009fb0 tx=0x7f817c009b30).stop 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- 172.21.15.195:0/4197827333 shutdown_connections 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 --2- 172.21.15.195:0/4197827333 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81880f66c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- 172.21.15.195:0/4197827333 >> 172.21.15.195:0/4197827333 conn(0x7f81880f1aa0 msgr2=0x7f81880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- 172.21.15.195:0/4197827333 shutdown_connections 2024-02-20T00:36:47.787 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- 172.21.15.195:0/4197827333 wait complete. 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 Processor -- start 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.750+0000 7f818dbe9700 1 -- start start 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818dbe9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818dbe9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8188106ac0 con 0x7f81880f62a0 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818cbe7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:47.788 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818cbe7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55084/0 (socket says 172.21.15.195:55084) 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818cbe7700 1 -- 172.21.15.195:0/2585067955 learned_addr learned my addr 172.21.15.195:0/2585067955 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818cbe7700 1 -- 172.21.15.195:0/2585067955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c005040 con 0x7f81880f62a0 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f818cbe7700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f817c0089f0 tx=0x7f817c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.751+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817c009320 con 0x7f81880f62a0 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8188106cc0 con 0x7f81880f62a0 2024-02-20T00:36:47.789 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f817c013380 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817c014e20 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8188107150 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f817c014420 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8188045ad0 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.752+0000 7f81857fa700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 0x7f81780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.753+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f817c040fd0 con 0x7f81880f62a0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.753+0000 7f8187fff700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 0x7f81780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.753+0000 7f8187fff700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 0x7f81780323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f81740096a0 tx=0x7f8174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:47.790 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.757+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f817c06d2b0 con 0x7f81880f62a0 2024-02-20T00:36:47.967 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.965+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8188108000 con 0x7f81880f62a0 2024-02-20T00:36:47.968 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.966+0000 7f81857fa700 1 -- 172.21.15.195:0/2585067955 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f817c012330 con 0x7f81880f62a0 2024-02-20T00:36:47.968 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:47.968 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:47.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.967+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 msgr2=0x7f81780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:47.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.967+0000 7f818dbe9700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 0x7f81780323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f81740096a0 tx=0x7f8174006b40).stop 2024-02-20T00:36:47.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.967+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 msgr2=0x7f81881065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:47.969 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.967+0000 7f818dbe9700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f817c0089f0 tx=0x7f817c00b040).stop 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 shutdown_connections 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f817802ff10 0x7f81780323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 --2- 172.21.15.195:0/2585067955 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81880f62a0 0x7f81881065d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 >> 172.21.15.195:0/2585067955 conn(0x7f81880f1aa0 msgr2=0x7f81880f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 shutdown_connections 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:47.968+0000 7f818dbe9700 1 -- 172.21.15.195:0/2585067955 wait complete. 2024-02-20T00:36:47.970 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:47.245273+0000 mgr.smithi087.iydsqo (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:48.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:47.967377+0000 mon.smithi087 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.195:0/2585067955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:49.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:49.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:50.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:49.245624+0000 mgr.smithi087.iydsqo (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 -- 172.21.15.195:0/2586369226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 msgr2=0x7f3e140f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 --2- 172.21.15.195:0/2586369226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e140f6850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f3e04009fb0 tx=0x7f3e04009b30).stop 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 -- 172.21.15.195:0/2586369226 shutdown_connections 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 --2- 172.21.15.195:0/2586369226 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e140f6850 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 -- 172.21.15.195:0/2586369226 >> 172.21.15.195:0/2586369226 conn(0x7f3e140f1b90 msgr2=0x7f3e140f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 -- 172.21.15.195:0/2586369226 shutdown_connections 2024-02-20T00:36:51.714 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.712+0000 7f3e18b42700 1 -- 172.21.15.195:0/2586369226 wait complete. 2024-02-20T00:36:51.715 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e18b42700 1 Processor -- start 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e18b42700 1 -- start start 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e18b42700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e18b42700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14106f70 con 0x7f3e140f6430 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e1259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e1259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55096/0 (socket says 172.21.15.195:55096) 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e1259c700 1 -- 172.21.15.195:0/1751998913 learned_addr learned my addr 172.21.15.195:0/1751998913 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e1259c700 1 -- 172.21.15.195:0/1751998913 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04005040 con 0x7f3e140f6430 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.713+0000 7f3e1259c700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3e04005010 tx=0x7f3e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e04009320 con 0x7f3e140f6430 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14107170 con 0x7f3e140f6430 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e04013380 con 0x7f3e140f6430 2024-02-20T00:36:51.716 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e04014e50 con 0x7f3e140f6430 2024-02-20T00:36:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14107600 con 0x7f3e140f6430 2024-02-20T00:36:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.715+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e040095f0 con 0x7f3e140f6430 2024-02-20T00:36:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.714+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14045ad0 con 0x7f3e140f6430 2024-02-20T00:36:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.715+0000 7f3e02ffd700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 0x7f3dfc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:51.717 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.715+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e04040e70 con 0x7f3e140f6430 2024-02-20T00:36:51.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.715+0000 7f3e11d9b700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 0x7f3dfc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:51.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.715+0000 7f3e11d9b700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 0x7f3dfc0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3e080096a0 tx=0x7f3e08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:51.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.719+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e04014420 con 0x7f3e140f6430 2024-02-20T00:36:51.924 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.923+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e1405ea50 con 0x7f3e140f6430 2024-02-20T00:36:51.925 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.923+0000 7f3e02ffd700 1 -- 172.21.15.195:0/1751998913 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e04010230 con 0x7f3e140f6430 2024-02-20T00:36:51.925 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:51.925 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 msgr2=0x7f3dfc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 0x7f3dfc0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3e080096a0 tx=0x7f3e08006b40).stop 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 msgr2=0x7f3e14106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3e04005010 tx=0x7f3e0400b040).stop 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 shutdown_connections 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f3dfc02ff10 0x7f3dfc0323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 --2- 172.21.15.195:0/1751998913 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e140f6430 0x7f3e14106a80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 >> 172.21.15.195:0/1751998913 conn(0x7f3e140f1b90 msgr2=0x7f3e140f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 shutdown_connections 2024-02-20T00:36:51.927 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:51.925+0000 7f3e18b42700 1 -- 172.21.15.195:0/1751998913 wait complete. 2024-02-20T00:36:51.928 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:51.245952+0000 mgr.smithi087.iydsqo (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:51.924673+0000 mon.smithi087 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.195:0/1751998913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:53.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:53.645 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:54.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:53.246213+0000 mgr.smithi087.iydsqo (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:55.501 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 -- 172.21.15.195:0/1491806839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 msgr2=0x7f72740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1491806839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72740f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f7264009fb0 tx=0x7f7264009b30).stop 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 -- 172.21.15.195:0/1491806839 shutdown_connections 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1491806839 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72740f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 -- 172.21.15.195:0/1491806839 >> 172.21.15.195:0/1491806839 conn(0x7f72740f1c70 msgr2=0x7f72740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 -- 172.21.15.195:0/1491806839 shutdown_connections 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.473+0000 7f727c2a3700 1 -- 172.21.15.195:0/1491806839 wait complete. 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727c2a3700 1 Processor -- start 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727c2a3700 1 -- start start 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727c2a3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727c2a3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7274106cd0 con 0x7f72740f6510 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727a03f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727a03f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:55120/0 (socket says 172.21.15.195:55120) 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727a03f700 1 -- 172.21.15.195:0/1506926070 learned_addr learned my addr 172.21.15.195:0/1506926070 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:55.502 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727a03f700 1 -- 172.21.15.195:0/1506926070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264005040 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.474+0000 7f727a03f700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f7264005010 tx=0x7f726400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7264009320 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7274106ed0 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7264013380 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7264014e50 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7274107360 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7264014420 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7274045ad0 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.475+0000 7f726affd700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 0x7f72600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.476+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7264040eb0 con 0x7f72740f6510 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.476+0000 7f727983e700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 0x7f72600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.476+0000 7f727983e700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 0x7f72600323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f72700096a0 tx=0x7f7270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:55.503 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.480+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72640095f0 con 0x7f72740f6510 2024-02-20T00:36:55.687 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.686+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7274108210 con 0x7f72740f6510 2024-02-20T00:36:55.688 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.686+0000 7f726affd700 1 -- 172.21.15.195:0/1506926070 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f726401db70 con 0x7f72740f6510 2024-02-20T00:36:55.688 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:55.688 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:55.689 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 msgr2=0x7f72600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:55.689 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 0x7f72600323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f72700096a0 tx=0x7f7270006b40).stop 2024-02-20T00:36:55.689 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 msgr2=0x7f72741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f7264005010 tx=0x7f726400b040).stop 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 shutdown_connections 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f726002ff10 0x7f72600323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 --2- 172.21.15.195:0/1506926070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72740f6510 0x7f72741067e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 >> 172.21.15.195:0/1506926070 conn(0x7f72740f1c70 msgr2=0x7f72740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 shutdown_connections 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:55.688+0000 7f727c2a3700 1 -- 172.21.15.195:0/1506926070 wait complete. 2024-02-20T00:36:55.690 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:36:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:55.246479+0000 mgr.smithi087.iydsqo (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:56.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:55.687537+0000 mon.smithi087 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.195:0/1506926070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:36:57.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:36:57.373 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:36:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:56.675666+0000 mon.smithi087 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:36:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:56.675892+0000 mon.smithi087 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:36:58.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:36:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:57.246868+0000 mgr.smithi087.iydsqo (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:36:59.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.361+0000 7ff80273f700 1 -- 172.21.15.195:0/1301508417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 msgr2=0x7ff7fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:59.365 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.361+0000 7ff80273f700 1 --2- 172.21.15.195:0/1301508417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc0f6930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7ff7ec009fb0 tx=0x7ff7ec009b30).stop 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 -- 172.21.15.195:0/1301508417 shutdown_connections 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 --2- 172.21.15.195:0/1301508417 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc0f6930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 -- 172.21.15.195:0/1301508417 >> 172.21.15.195:0/1301508417 conn(0x7ff7fc0f1c70 msgr2=0x7ff7fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 -- 172.21.15.195:0/1301508417 shutdown_connections 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 -- 172.21.15.195:0/1301508417 wait complete. 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 Processor -- start 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.362+0000 7ff80273f700 1 -- start start 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff80273f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff80273f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc106cd0 con 0x7ff7fc0f6510 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff7fbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff7fbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38366/0 (socket says 172.21.15.195:38366) 2024-02-20T00:36:59.366 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff7fbfff700 1 -- 172.21.15.195:0/2551979700 learned_addr learned my addr 172.21.15.195:0/2551979700 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff7fbfff700 1 -- 172.21.15.195:0/2551979700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec005040 con 0x7ff7fc0f6510 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.363+0000 7ff7fbfff700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff7ec005010 tx=0x7ff7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.364+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7ec009320 con 0x7ff7fc0f6510 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.364+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc106ed0 con 0x7ff7fc0f6510 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.364+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7ec013380 con 0x7ff7fc0f6510 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.364+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7ec014e50 con 0x7ff7fc0f6510 2024-02-20T00:36:59.367 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.364+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc107360 con 0x7ff7fc0f6510 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7ec0095f0 con 0x7ff7fc0f6510 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc045ad0 con 0x7ff7fc0f6510 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff7f8ff9700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 0x7ff7e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7ec040e50 con 0x7ff7fc0f6510 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff7fb7fe700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 0x7ff7e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:36:59.368 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.365+0000 7ff7fb7fe700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 0x7ff7e40323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff7f00096a0 tx=0x7ff7f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:36:59.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.369+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7ec014420 con 0x7ff7fc0f6510 2024-02-20T00:36:59.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.571+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7fc108210 con 0x7ff7fc0f6510 2024-02-20T00:36:59.573 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.572+0000 7ff7f8ff9700 1 -- 172.21.15.195:0/2551979700 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7ec014420 con 0x7ff7fc0f6510 2024-02-20T00:36:59.574 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:36:59.574 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:36:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 msgr2=0x7ff7e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 0x7ff7e40323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff7f00096a0 tx=0x7ff7f0006b40).stop 2024-02-20T00:36:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 msgr2=0x7ff7fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:36:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7ff7ec005010 tx=0x7ff7ec00b040).stop 2024-02-20T00:36:59.576 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 shutdown_connections 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ff7e402ff10 0x7ff7e40323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 --2- 172.21.15.195:0/2551979700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff7fc0f6510 0x7ff7fc1067e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.574+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 >> 172.21.15.195:0/2551979700 conn(0x7ff7fc0f1c70 msgr2=0x7ff7fc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.575+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 shutdown_connections 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:36:59.575+0000 7ff80273f700 1 -- 172.21.15.195:0/2551979700 wait complete. 2024-02-20T00:36:59.577 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:36:59.247275+0000 mgr.smithi087.iydsqo (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:36:59.573222+0000 mon.smithi087 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.195:0/2551979700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:01.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:01.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:02.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:02 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:01.247685+0000 mgr.smithi087.iydsqo (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.091+0000 7f4bd9b24700 1 -- 172.21.15.195:0/2331909978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 msgr2=0x7f4bd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.091+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/2331909978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd40f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4bc4009fb0 tx=0x7f4bc4009b30).stop 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.091+0000 7f4bd9b24700 1 -- 172.21.15.195:0/2331909978 shutdown_connections 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.091+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/2331909978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd40f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.091+0000 7f4bd9b24700 1 -- 172.21.15.195:0/2331909978 >> 172.21.15.195:0/2331909978 conn(0x7f4bd40f0b70 msgr2=0x7f4bd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 -- 172.21.15.195:0/2331909978 shutdown_connections 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 -- 172.21.15.195:0/2331909978 wait complete. 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 Processor -- start 2024-02-20T00:37:03.095 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 -- start start 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.092+0000 7f4bd9b24700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4106ce0 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:38390/0 (socket says 172.21.15.195:38390) 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd37fe700 1 -- 172.21.15.195:0/1980888339 learned_addr learned my addr 172.21.15.195:0/1980888339 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd37fe700 1 -- 172.21.15.195:0/1980888339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4005040 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd37fe700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4bc4005010 tx=0x7f4bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc4009320 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd4106ee0 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bc4013380 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc4014e50 con 0x7f4bd40f5410 2024-02-20T00:37:03.096 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.093+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd4107370 con 0x7f4bd40f5410 2024-02-20T00:37:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.094+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bc40095f0 con 0x7f4bd40f5410 2024-02-20T00:37:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.094+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd4045ad0 con 0x7f4bd40f5410 2024-02-20T00:37:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.094+0000 7f4bbbfff700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 0x7f4bbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:03.097 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.094+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bc4040fa0 con 0x7f4bd40f5410 2024-02-20T00:37:03.098 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.094+0000 7f4bd2ffd700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 0x7f4bbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:03.098 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.095+0000 7f4bd2ffd700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 0x7f4bbc0323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4bc80096a0 tx=0x7f4bc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:03.100 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.099+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bc4014420 con 0x7f4bd40f5410 2024-02-20T00:37:03.303 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.302+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bd4108220 con 0x7f4bd40f5410 2024-02-20T00:37:03.304 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.303+0000 7f4bbbfff700 1 -- 172.21.15.195:0/1980888339 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bc4010230 con 0x7f4bd40f5410 2024-02-20T00:37:03.304 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:03.304 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 msgr2=0x7f4bbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 0x7f4bbc0323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4bc80096a0 tx=0x7f4bc8006b40).stop 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 msgr2=0x7f4bd41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4bc4005010 tx=0x7f4bc400b040).stop 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 shutdown_connections 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bbc02ff10 0x7f4bbc0323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 --2- 172.21.15.195:0/1980888339 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd40f5410 0x7f4bd41067f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 >> 172.21.15.195:0/1980888339 conn(0x7f4bd40f0b70 msgr2=0x7f4bd405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 shutdown_connections 2024-02-20T00:37:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:03.304+0000 7f4bd9b24700 1 -- 172.21.15.195:0/1980888339 wait complete. 2024-02-20T00:37:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:03.247983+0000 mgr.smithi087.iydsqo (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:04.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:04 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:03.303865+0000 mon.smithi087 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.195:0/1980888339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:05.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:05.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:06.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:06 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:05.248275+0000 mgr.smithi087.iydsqo (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:07.156 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 -- 172.21.15.195:0/1690802540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 msgr2=0x7ffbd40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/1690802540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd40f7d00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7ffbc4009fb0 tx=0x7ffbc4009b30).stop 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 -- 172.21.15.195:0/1690802540 shutdown_connections 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/1690802540 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd40f7d00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 -- 172.21.15.195:0/1690802540 >> 172.21.15.195:0/1690802540 conn(0x7ffbd40f0120 msgr2=0x7ffbd40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 -- 172.21.15.195:0/1690802540 shutdown_connections 2024-02-20T00:37:07.157 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.155+0000 7ffbd3fff700 1 -- 172.21.15.195:0/1690802540 wait complete. 2024-02-20T00:37:07.158 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd3fff700 1 Processor -- start 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd3fff700 1 -- start start 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd3fff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd4106af0 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd2ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46306/0 (socket says 172.21.15.195:46306) 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd2ffd700 1 -- 172.21.15.195:0/2300572493 learned_addr learned my addr 172.21.15.195:0/2300572493 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.156+0000 7ffbd2ffd700 1 -- 172.21.15.195:0/2300572493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc4005040 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbd2ffd700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ffbc40089f0 tx=0x7ffbc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbc4009320 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd4106cf0 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd4107180 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbc4013380 con 0x7ffbd40f78e0 2024-02-20T00:37:07.159 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.157+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbc4014e40 con 0x7ffbd40f78e0 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.158+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbc40263a0 con 0x7ffbd40f78e0 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.158+0000 7ffbbb7fe700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.158+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4004fa0 con 0x7ffbd40f78e0 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.158+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbc4040e10 con 0x7ffbd40f78e0 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.158+0000 7ffbd27fc700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:07.161 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.159+0000 7ffbd27fc700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:07.164 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.163+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbc4012550 con 0x7ffbd40f78e0 2024-02-20T00:37:07.370 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.369+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbb4005670 con 0x7ffbd40f78e0 2024-02-20T00:37:07.371 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.369+0000 7ffbbb7fe700 1 -- 172.21.15.195:0/2300572493 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffbc4012370 con 0x7ffbd40f78e0 2024-02-20T00:37:07.371 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:07.371 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 msgr2=0x7ffbbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).stop 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 msgr2=0x7ffbd4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ffbc40089f0 tx=0x7ffbc400b040).stop 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 shutdown_connections 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 --2- 172.21.15.195:0/2300572493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbd40f78e0 0x7ffbd4106600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 >> 172.21.15.195:0/2300572493 conn(0x7ffbd40f0120 msgr2=0x7ffbd40f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 shutdown_connections 2024-02-20T00:37:07.373 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:07.371+0000 7ffbd3fff700 1 -- 172.21.15.195:0/2300572493 wait complete. 2024-02-20T00:37:07.374 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:08.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:07.248650+0000 mgr.smithi087.iydsqo (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:08.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:08 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:07.370613+0000 mon.smithi087 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.195:0/2300572493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:08.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:08.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:10.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:10 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:09.248936+0000 mgr.smithi087.iydsqo (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:10.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 -- 172.21.15.195:0/48765596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 msgr2=0x7fc1cc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:10.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/48765596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc0f4be0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fc1c0009fb0 tx=0x7fc1c0009b30).stop 2024-02-20T00:37:10.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 -- 172.21.15.195:0/48765596 shutdown_connections 2024-02-20T00:37:10.817 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/48765596 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc0f4be0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 -- 172.21.15.195:0/48765596 >> 172.21.15.195:0/48765596 conn(0x7fc1cc0f02e0 msgr2=0x7fc1cc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 -- 172.21.15.195:0/48765596 shutdown_connections 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 -- 172.21.15.195:0/48765596 wait complete. 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.815+0000 7fc1d2a61700 1 Processor -- start 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1d2a61700 1 -- start start 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1d2a61700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:10.818 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1d2a61700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc106cc0 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1cbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1cbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46328/0 (socket says 172.21.15.195:46328) 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1cbfff700 1 -- 172.21.15.195:0/2409492771 learned_addr learned my addr 172.21.15.195:0/2409492771 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1cbfff700 1 -- 172.21.15.195:0/2409492771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0005040 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.816+0000 7fc1cbfff700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0009320 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc106ec0 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1c0013380 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0014e50 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.820 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc107350 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.817+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1c00263a0 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.818+0000 7fc1c8ff9700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.818+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1c0040f80 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.818+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc045ad0 con 0x7fc1cc0f47c0 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.818+0000 7fc1cb7fe700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:10.821 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.818+0000 7fc1cb7fe700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:10.824 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:10.823+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1c0012920 con 0x7fc1cc0f47c0 2024-02-20T00:37:11.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.027+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1cc0fc180 con 0x7fc1cc0f47c0 2024-02-20T00:37:11.028 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.028+0000 7fc1c8ff9700 1 -- 172.21.15.195:0/2409492771 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1c0012330 con 0x7fc1cc0f47c0 2024-02-20T00:37:11.028 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:11.028 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 msgr2=0x7fc1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).stop 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 msgr2=0x7fc1cc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).stop 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 shutdown_connections 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 --2- 172.21.15.195:0/2409492771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:11.030 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 >> 172.21.15.195:0/2409492771 conn(0x7fc1cc0f02e0 msgr2=0x7fc1cc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:11.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 shutdown_connections 2024-02-20T00:37:11.031 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:11.029+0000 7fc1d2a61700 1 -- 172.21.15.195:0/2409492771 wait complete. 2024-02-20T00:37:11.031 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:11.783 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:11 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:11.027955+0000 mon.smithi087 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.195:0/2409492771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:12.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:12.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:12.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:12 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:11.249272+0000 mgr.smithi087.iydsqo (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:14.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:14 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:13.249553+0000 mgr.smithi087.iydsqo (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:14.650 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 -- 172.21.15.195:0/3290463470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 msgr2=0x7f86800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 --2- 172.21.15.195:0/3290463470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86800f55c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f8670009fb0 tx=0x7f8670009b30).stop 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 -- 172.21.15.195:0/3290463470 shutdown_connections 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 --2- 172.21.15.195:0/3290463470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86800f55c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 -- 172.21.15.195:0/3290463470 >> 172.21.15.195:0/3290463470 conn(0x7f86800f09a0 msgr2=0x7f86800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 -- 172.21.15.195:0/3290463470 shutdown_connections 2024-02-20T00:37:14.651 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.644+0000 7f867f59e700 1 -- 172.21.15.195:0/3290463470 wait complete. 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867f59e700 1 Processor -- start 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867f59e700 1 -- start start 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867f59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867f59e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680106ac0 con 0x7f86800f51a0 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:46354/0 (socket says 172.21.15.195:46354) 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867e59c700 1 -- 172.21.15.195:0/185235965 learned_addr learned my addr 172.21.15.195:0/185235965 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:14.652 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867e59c700 1 -- 172.21.15.195:0/185235965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8670005040 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.645+0000 7f867e59c700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f8670009830 tx=0x7f867000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8670009320 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8680106cc0 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8670013380 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8670014420 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8680107150 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f867001e420 con 0x7f86800f51a0 2024-02-20T00:37:14.653 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.646+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8680045ad0 con 0x7f86800f51a0 2024-02-20T00:37:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.647+0000 7f866effd700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 0x7f8668032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.647+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8670040e30 con 0x7f86800f51a0 2024-02-20T00:37:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.647+0000 7f867dd9b700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 0x7f8668032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:14.654 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.647+0000 7f867dd9b700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 0x7f8668032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f86740096a0 tx=0x7f8674006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:14.655 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.652+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f867001cb10 con 0x7f86800f51a0 2024-02-20T00:37:14.855 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.854+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86800fad30 con 0x7f86800f51a0 2024-02-20T00:37:14.856 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.855+0000 7f866effd700 1 -- 172.21.15.195:0/185235965 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8670004490 con 0x7f86800f51a0 2024-02-20T00:37:14.856 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:14.856 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 msgr2=0x7f8668032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 0x7f8668032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f86740096a0 tx=0x7f8674006b40).stop 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 msgr2=0x7f86801065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f8670009830 tx=0x7f867000b040).stop 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 shutdown_connections 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f866802ff60 0x7f8668032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 --2- 172.21.15.195:0/185235965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86800f51a0 0x7f86801065d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 >> 172.21.15.195:0/185235965 conn(0x7f86800f09a0 msgr2=0x7f868005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:14.858 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 shutdown_connections 2024-02-20T00:37:14.859 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:14.857+0000 7f867f59e700 1 -- 172.21.15.195:0/185235965 wait complete. 2024-02-20T00:37:14.859 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:15.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:15 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:14.855493+0000 mon.smithi087 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.195:0/185235965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:16.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:16 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:15.249881+0000 mgr.smithi087.iydsqo (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:16.544 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:16.544 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 -- 172.21.15.195:0/3098211139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 msgr2=0x7f29880f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 --2- 172.21.15.195:0/3098211139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29880f4be0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f297c009fb0 tx=0x7f297c009b30).stop 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 -- 172.21.15.195:0/3098211139 shutdown_connections 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 --2- 172.21.15.195:0/3098211139 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29880f4be0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 -- 172.21.15.195:0/3098211139 >> 172.21.15.195:0/3098211139 conn(0x7f29880f02e0 msgr2=0x7f29880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 -- 172.21.15.195:0/3098211139 shutdown_connections 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.443+0000 7f298d8e4700 1 -- 172.21.15.195:0/3098211139 wait complete. 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.444+0000 7f298d8e4700 1 Processor -- start 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.444+0000 7f298d8e4700 1 -- start start 2024-02-20T00:37:18.447 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f298d8e4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:18.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f298d8e4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2988106cc0 con 0x7f29880f47c0 2024-02-20T00:37:18.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f2986ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:18.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f2986ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41872/0 (socket says 172.21.15.195:41872) 2024-02-20T00:37:18.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f2986ffd700 1 -- 172.21.15.195:0/2398670176 learned_addr learned my addr 172.21.15.195:0/2398670176 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:18.448 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f2986ffd700 1 -- 172.21.15.195:0/2398670176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c005040 con 0x7f29880f47c0 2024-02-20T00:37:18.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.445+0000 7f2986ffd700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f297c005010 tx=0x7f297c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:18.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.446+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c009320 con 0x7f29880f47c0 2024-02-20T00:37:18.449 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.446+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f297c013380 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.446+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2988106ec0 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.446+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c014e50 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.446+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2988107350 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f297c014420 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2988045ad0 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f296f7fe700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 0x7f29700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f297c040f70 con 0x7f29880f47c0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f29867fc700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 0x7f29700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:18.450 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.447+0000 7f29867fc700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 0x7f29700323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f29780096a0 tx=0x7f2978006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:18.453 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.451+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f297c0095f0 con 0x7f29880f47c0 2024-02-20T00:37:18.658 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.656+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29881081a0 con 0x7f29880f47c0 2024-02-20T00:37:18.659 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.657+0000 7f296f7fe700 1 -- 172.21.15.195:0/2398670176 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f297c01db70 con 0x7f29880f47c0 2024-02-20T00:37:18.659 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:18.659 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:18.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 msgr2=0x7f29700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:18.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 0x7f29700323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f29780096a0 tx=0x7f2978006b40).stop 2024-02-20T00:37:18.660 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 msgr2=0x7f29881067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f297c005010 tx=0x7f297c00b040).stop 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 shutdown_connections 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f297002ff10 0x7f29700323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 --2- 172.21.15.195:0/2398670176 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29880f47c0 0x7f29881067d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 >> 172.21.15.195:0/2398670176 conn(0x7f29880f02e0 msgr2=0x7f29880f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 shutdown_connections 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:18.659+0000 7f298d8e4700 1 -- 172.21.15.195:0/2398670176 wait complete. 2024-02-20T00:37:18.661 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:18.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:18 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:17.250191+0000 mgr.smithi087.iydsqo (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:19.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:19 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:18.658421+0000 mon.smithi087 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.195:0/2398670176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:20.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:20.397 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:20.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:20 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:19.250558+0000 mgr.smithi087.iydsqo (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- 172.21.15.195:0/1331993475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 msgr2=0x7f7f7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 --2- 172.21.15.195:0/1331993475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f7f6c009fb0 tx=0x7f7f6c009b30).stop 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- 172.21.15.195:0/1331993475 shutdown_connections 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 --2- 172.21.15.195:0/1331993475 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- 172.21.15.195:0/1331993475 >> 172.21.15.195:0/1331993475 conn(0x7f7f7c0f0b70 msgr2=0x7f7f7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- 172.21.15.195:0/1331993475 shutdown_connections 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- 172.21.15.195:0/1331993475 wait complete. 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 Processor -- start 2024-02-20T00:37:22.410 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.408+0000 7f7f8248a700 1 -- start start 2024-02-20T00:37:22.411 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f8248a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f8248a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c105bd0 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f7bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f7bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41888/0 (socket says 172.21.15.195:41888) 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f7bfff700 1 -- 172.21.15.195:0/3434068842 learned_addr learned my addr 172.21.15.195:0/3434068842 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f7bfff700 1 -- 172.21.15.195:0/3434068842 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c005040 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.409+0000 7f7f7bfff700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7f6c005010 tx=0x7f7f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.410+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6c009320 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.410+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f6c013380 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.410+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c105dd0 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.410+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6c014e50 con 0x7f7f7c0f5410 2024-02-20T00:37:22.412 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.410+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c106260 con 0x7f7f7c0f5410 2024-02-20T00:37:22.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f6c0095f0 con 0x7f7f7c0f5410 2024-02-20T00:37:22.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c045ad0 con 0x7f7f7c0f5410 2024-02-20T00:37:22.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f78ff9700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 0x7f7f640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:22.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f6c040ee0 con 0x7f7f7c0f5410 2024-02-20T00:37:22.413 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f7b7fe700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 0x7f7f640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:22.414 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.411+0000 7f7f7b7fe700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 0x7f7f640323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7f700096a0 tx=0x7f7f70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:22.417 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.416+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f6c014420 con 0x7f7f7c0f5410 2024-02-20T00:37:22.623 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.622+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f7c107110 con 0x7f7f7c0f5410 2024-02-20T00:37:22.624 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.623+0000 7f7f78ff9700 1 -- 172.21.15.195:0/3434068842 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f6c01d980 con 0x7f7f7c0f5410 2024-02-20T00:37:22.624 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:22.625 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:22.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 msgr2=0x7f7f640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:22.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 0x7f7f640323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7f700096a0 tx=0x7f7f70006b40).stop 2024-02-20T00:37:22.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 msgr2=0x7f7f7c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:22.626 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7f6c005010 tx=0x7f7f6c00b040).stop 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 shutdown_connections 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f7f6402ff10 0x7f7f640323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 --2- 172.21.15.195:0/3434068842 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7f7c0f5410 0x7f7f7c1056e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.624+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 >> 172.21.15.195:0/3434068842 conn(0x7f7f7c0f0b70 msgr2=0x7f7f7c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.625+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 shutdown_connections 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:22.625+0000 7f7f8248a700 1 -- 172.21.15.195:0/3434068842 wait complete. 2024-02-20T00:37:22.627 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:22.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:22 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:21.250869+0000 mgr.smithi087.iydsqo (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:23.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:23 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:22.623691+0000 mon.smithi087 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.195:0/3434068842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:24.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:24.218 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:24.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:24 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:23.251206+0000 mgr.smithi087.iydsqo (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:26.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 -- 172.21.15.195:0/1168781255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 msgr2=0x7fba500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:26.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 --2- 172.21.15.195:0/1168781255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba500f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fba44009fb0 tx=0x7fba44009b30).stop 2024-02-20T00:37:26.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 -- 172.21.15.195:0/1168781255 shutdown_connections 2024-02-20T00:37:26.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 --2- 172.21.15.195:0/1168781255 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba500f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:26.059 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 -- 172.21.15.195:0/1168781255 >> 172.21.15.195:0/1168781255 conn(0x7fba500f0b70 msgr2=0x7fba500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:26.059 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 -- 172.21.15.195:0/1168781255 shutdown_connections 2024-02-20T00:37:26.059 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.056+0000 7fba570f4700 1 -- 172.21.15.195:0/1168781255 wait complete. 2024-02-20T00:37:26.059 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba570f4700 1 Processor -- start 2024-02-20T00:37:26.059 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba570f4700 1 -- start start 2024-02-20T00:37:26.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba570f4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:26.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba570f4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba50060240 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba54e90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba54e90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:41904/0 (socket says 172.21.15.195:41904) 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.057+0000 7fba54e90700 1 -- 172.21.15.195:0/389372673 learned_addr learned my addr 172.21.15.195:0/389372673 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba54e90700 1 -- 172.21.15.195:0/389372673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44005040 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba54e90700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fba44005010 tx=0x7fba4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba44009320 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba44013380 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba50060440 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba44014e50 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.058+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba50109b90 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.059+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba440095f0 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.059+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba50045ad0 con 0x7fba500f5410 2024-02-20T00:37:26.061 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.059+0000 7fba4d7fa700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 0x7fba400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:26.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.059+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba44040fa0 con 0x7fba500f5410 2024-02-20T00:37:26.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.059+0000 7fba4ffff700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 0x7fba400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:26.062 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.060+0000 7fba4ffff700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 0x7fba400323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fba3c0096a0 tx=0x7fba3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:26.065 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.064+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba44014420 con 0x7fba500f5410 2024-02-20T00:37:26.269 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.267+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba5010a3b0 con 0x7fba500f5410 2024-02-20T00:37:26.269 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.268+0000 7fba4d7fa700 1 -- 172.21.15.195:0/389372673 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba44010230 con 0x7fba500f5410 2024-02-20T00:37:26.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:26.270 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 msgr2=0x7fba400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 0x7fba400323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fba3c0096a0 tx=0x7fba3c006b40).stop 2024-02-20T00:37:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 msgr2=0x7fba501094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fba44005010 tx=0x7fba4400b040).stop 2024-02-20T00:37:26.271 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 shutdown_connections 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fba4002ff10 0x7fba400323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 --2- 172.21.15.195:0/389372673 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fba500f5410 0x7fba501094e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 >> 172.21.15.195:0/389372673 conn(0x7fba500f0b70 msgr2=0x7fba500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 shutdown_connections 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:26.270+0000 7fba570f4700 1 -- 172.21.15.195:0/389372673 wait complete. 2024-02-20T00:37:26.272 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:26.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:25.251581+0000 mgr.smithi087.iydsqo (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:26.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:26 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:26.269338+0000 mon.smithi087 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.195:0/389372673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:27.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:27.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:28.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:28 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:27.251839+0000 mgr.smithi087.iydsqo (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:29.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- 172.21.15.195:0/685254467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 msgr2=0x7fefe40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:29.848 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 --2- 172.21.15.195:0/685254467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe40f8f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fefe0009fb0 tx=0x7fefe0009b30).stop 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- 172.21.15.195:0/685254467 shutdown_connections 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 --2- 172.21.15.195:0/685254467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe40f8f20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- 172.21.15.195:0/685254467 >> 172.21.15.195:0/685254467 conn(0x7fefe40f09a0 msgr2=0x7fefe40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- 172.21.15.195:0/685254467 shutdown_connections 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- 172.21.15.195:0/685254467 wait complete. 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 Processor -- start 2024-02-20T00:37:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- start start 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.844+0000 7fefeb542700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4106af0 con 0x7fefe40601e0 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefea540700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefea540700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:50400/0 (socket says 172.21.15.195:50400) 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefea540700 1 -- 172.21.15.195:0/3318064438 learned_addr learned my addr 172.21.15.195:0/3318064438 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefea540700 1 -- 172.21.15.195:0/3318064438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe0005040 con 0x7fefe40601e0 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefea540700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fefe0005010 tx=0x7fefe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe0009320 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe4106cf0 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe0013380 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe4107180 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.845+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe0014e50 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.846+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe4045ad0 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.846+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefe00095f0 con 0x7fefe40601e0 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.846+0000 7fefdaffd700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 0x7fefd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:29.851 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.846+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefe0040ff0 con 0x7fefe40601e0 2024-02-20T00:37:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.846+0000 7fefe9d3f700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 0x7fefd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.847+0000 7fefe9d3f700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 0x7fefd00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fefd40096a0 tx=0x7fefd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:29.853 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:29.851+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefe006d2b0 con 0x7fefe40601e0 2024-02-20T00:37:30.057 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.055+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefe4107fd0 con 0x7fefe40601e0 2024-02-20T00:37:30.058 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.056+0000 7fefdaffd700 1 -- 172.21.15.195:0/3318064438 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefe0014420 con 0x7fefe40601e0 2024-02-20T00:37:30.058 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:30.058 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 msgr2=0x7fefd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 0x7fefd00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fefd40096a0 tx=0x7fefd4006b40).stop 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 msgr2=0x7fefe4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fefe0005010 tx=0x7fefe000b040).stop 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 shutdown_connections 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fefd002ff10 0x7fefd00323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 --2- 172.21.15.195:0/3318064438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefe40601e0 0x7fefe4106600 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 >> 172.21.15.195:0/3318064438 conn(0x7fefe40f09a0 msgr2=0x7fefe40f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 shutdown_connections 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:30.058+0000 7fefeb542700 1 -- 172.21.15.195:0/3318064438 wait complete. 2024-02-20T00:37:30.060 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:29.252137+0000 mgr.smithi087.iydsqo (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:30.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:30 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:30.057287+0000 mon.smithi087 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.195:0/3318064438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:31.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:31.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:32.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:32 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:31.252490+0000 mgr.smithi087.iydsqo (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- 172.21.15.195:0/457301246 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 msgr2=0x7fd3cc0f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/457301246 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc0f6ee0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fd3c8009fb0 tx=0x7fd3c8009b30).stop 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- 172.21.15.195:0/457301246 shutdown_connections 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/457301246 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc0f6ee0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- 172.21.15.195:0/457301246 >> 172.21.15.195:0/457301246 conn(0x7fd3cc0f0b70 msgr2=0x7fd3cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- 172.21.15.195:0/457301246 shutdown_connections 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- 172.21.15.195:0/457301246 wait complete. 2024-02-20T00:37:33.536 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 Processor -- start 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- start start 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.515+0000 7fd3d3c81700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc10d2c0 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d1a1d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d1a1d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:50416/0 (socket says 172.21.15.195:50416) 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d1a1d700 1 -- 172.21.15.195:0/122845049 learned_addr learned my addr 172.21.15.195:0/122845049 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d1a1d700 1 -- 172.21.15.195:0/122845049 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c8005040 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d1a1d700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fd3c80089f0 tx=0x7fd3c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c8009320 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc10d4c0 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3c8013380 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.516+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3c8014e20 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.517+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc10e100 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.517+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3c80263a0 con 0x7fd3cc060280 2024-02-20T00:37:33.537 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.517+0000 7fd3c27fc700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 0x7fd3b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:33.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.517+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc10d950 con 0x7fd3cc060280 2024-02-20T00:37:33.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.517+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3c8041270 con 0x7fd3cc060280 2024-02-20T00:37:33.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.518+0000 7fd3d121c700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 0x7fd3b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:33.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.518+0000 7fd3d121c700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 0x7fd3b80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd3bc0096a0 tx=0x7fd3bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:33.538 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.523+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3c8014600 con 0x7fd3cc060280 2024-02-20T00:37:33.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.727+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3cc059100 con 0x7fd3cc060280 2024-02-20T00:37:33.729 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.728+0000 7fd3c27fc700 1 -- 172.21.15.195:0/122845049 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3c8014420 con 0x7fd3cc060280 2024-02-20T00:37:33.730 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:33.730 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:33.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 msgr2=0x7fd3b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:33.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 0x7fd3b80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd3bc0096a0 tx=0x7fd3bc006b40).stop 2024-02-20T00:37:33.731 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 msgr2=0x7fd3cc10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fd3c80089f0 tx=0x7fd3c800b040).stop 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 shutdown_connections 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fd3b802ff10 0x7fd3b80323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 --2- 172.21.15.195:0/122845049 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd3cc060280 0x7fd3cc10cdd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 >> 172.21.15.195:0/122845049 conn(0x7fd3cc0f0b70 msgr2=0x7fd3cc0f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 shutdown_connections 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:33.730+0000 7fd3d3c81700 1 -- 172.21.15.195:0/122845049 wait complete. 2024-02-20T00:37:33.732 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:33.252879+0000 mgr.smithi087.iydsqo (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:34.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:34 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:33.729118+0000 mon.smithi087 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.195:0/122845049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:35.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:35.420 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:36.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:36 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:35.253207+0000 mgr.smithi087.iydsqo (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 -- 172.21.15.195:0/242285373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 msgr2=0x7f4bd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/242285373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd00f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f4bc4009fb0 tx=0x7f4bc4009b30).stop 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 -- 172.21.15.195:0/242285373 shutdown_connections 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/242285373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd00f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 -- 172.21.15.195:0/242285373 >> 172.21.15.195:0/242285373 conn(0x7f4bd00f0b70 msgr2=0x7f4bd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 -- 172.21.15.195:0/242285373 shutdown_connections 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 -- 172.21.15.195:0/242285373 wait complete. 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.396+0000 7f4bd64c1700 1 Processor -- start 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bd64c1700 1 -- start start 2024-02-20T00:37:37.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bd64c1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bd64c1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd0060240 con 0x7f4bd00f5410 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bcffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bcffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34046/0 (socket says 172.21.15.195:34046) 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bcffff700 1 -- 172.21.15.195:0/656742890 learned_addr learned my addr 172.21.15.195:0/656742890 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bcffff700 1 -- 172.21.15.195:0/656742890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4005040 con 0x7f4bd00f5410 2024-02-20T00:37:37.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.397+0000 7f4bcffff700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4bc4005010 tx=0x7f4bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:37.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc4009320 con 0x7f4bd00f5410 2024-02-20T00:37:37.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd0060440 con 0x7f4bd00f5410 2024-02-20T00:37:37.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bc4013380 con 0x7f4bd00f5410 2024-02-20T00:37:37.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc4014e50 con 0x7f4bd00f5410 2024-02-20T00:37:37.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd0109b90 con 0x7f4bd00f5410 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bc4014420 con 0x7f4bd00f5410 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.398+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd0045ad0 con 0x7f4bd00f5410 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.399+0000 7f4bccff9700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 0x7f4bb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.399+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bc4040f70 con 0x7f4bd00f5410 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.399+0000 7f4bcf7fe700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 0x7f4bb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:37.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.399+0000 7f4bcf7fe700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 0x7f4bb80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4bc00096a0 tx=0x7f4bc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:37.405 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.403+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bc40095f0 con 0x7f4bd00f5410 2024-02-20T00:37:37.609 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.608+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bd010a3a0 con 0x7f4bd00f5410 2024-02-20T00:37:37.610 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.609+0000 7f4bccff9700 1 -- 172.21.15.195:0/656742890 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bc401db70 con 0x7f4bd00f5410 2024-02-20T00:37:37.610 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:37.611 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:37.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 msgr2=0x7f4bb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:37.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 0x7f4bb80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4bc00096a0 tx=0x7f4bc0006b40).stop 2024-02-20T00:37:37.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 msgr2=0x7f4bd01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:37.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f4bc4005010 tx=0x7f4bc400b040).stop 2024-02-20T00:37:37.612 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 shutdown_connections 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f4bb802ff10 0x7f4bb80323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 --2- 172.21.15.195:0/656742890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4bd00f5410 0x7f4bd01094e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 >> 172.21.15.195:0/656742890 conn(0x7f4bd00f0b70 msgr2=0x7f4bd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 shutdown_connections 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:37.610+0000 7f4bd64c1700 1 -- 172.21.15.195:0/656742890 wait complete. 2024-02-20T00:37:37.613 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:37.253545+0000 mgr.smithi087.iydsqo (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:38.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:38 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:37.609838+0000 mon.smithi087 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.195:0/656742890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:39.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:39.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:40.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:40 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:39.253903+0000 mgr.smithi087.iydsqo (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:41.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 -- 172.21.15.195:0/3544405566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 msgr2=0x7fb68c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 --2- 172.21.15.195:0/3544405566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fb67c009fb0 tx=0x7fb67c009b30).stop 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 -- 172.21.15.195:0/3544405566 shutdown_connections 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 --2- 172.21.15.195:0/3544405566 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 -- 172.21.15.195:0/3544405566 >> 172.21.15.195:0/3544405566 conn(0x7fb68c0f0b70 msgr2=0x7fb68c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 -- 172.21.15.195:0/3544405566 shutdown_connections 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.396+0000 7fb693434700 1 -- 172.21.15.195:0/3544405566 wait complete. 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb693434700 1 Processor -- start 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb693434700 1 -- start start 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb693434700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb693434700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c106d00 con 0x7fb68c0f5410 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb6911d0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:41.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb6911d0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34064/0 (socket says 172.21.15.195:34064) 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.397+0000 7fb6911d0700 1 -- 172.21.15.195:0/2429332051 learned_addr learned my addr 172.21.15.195:0/2429332051 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb6911d0700 1 -- 172.21.15.195:0/2429332051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c005040 con 0x7fb68c0f5410 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb6911d0700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fb67c0089f0 tx=0x7fb67c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c009320 con 0x7fb68c0f5410 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb67c013380 con 0x7fb68c0f5410 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c106f00 con 0x7fb68c0f5410 2024-02-20T00:37:41.401 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67c01ed80 con 0x7fb68c0f5410 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.398+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c107390 con 0x7fb68c0f5410 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.399+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb67c014420 con 0x7fb68c0f5410 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.399+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c045ad0 con 0x7fb68c0f5410 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.399+0000 7fb681ffb700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 0x7fb678032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.399+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb67c041030 con 0x7fb68c0f5410 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.400+0000 7fb6909cf700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 0x7fb678032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:41.402 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.400+0000 7fb6909cf700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 0x7fb678032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb6880096a0 tx=0x7fb688006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:41.406 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.405+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb67c0095f0 con 0x7fb68c0f5410 2024-02-20T00:37:41.614 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.612+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb68c0fafa0 con 0x7fb68c0f5410 2024-02-20T00:37:41.615 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.613+0000 7fb681ffb700 1 -- 172.21.15.195:0/2429332051 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb67c01cb10 con 0x7fb68c0f5410 2024-02-20T00:37:41.615 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:41.615 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:41.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 msgr2=0x7fb678032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:41.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 0x7fb678032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb6880096a0 tx=0x7fb688006b40).stop 2024-02-20T00:37:41.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 msgr2=0x7fb68c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:41.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fb67c0089f0 tx=0x7fb67c00b040).stop 2024-02-20T00:37:41.617 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 shutdown_connections 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fb67802ffb0 0x7fb678032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 --2- 172.21.15.195:0/2429332051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb68c0f5410 0x7fb68c106810 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 >> 172.21.15.195:0/2429332051 conn(0x7fb68c0f0b70 msgr2=0x7fb68c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 shutdown_connections 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:41.615+0000 7fb693434700 1 -- 172.21.15.195:0/2429332051 wait complete. 2024-02-20T00:37:41.618 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:42.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:41.254369+0000 mgr.smithi087.iydsqo (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:42.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:42 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:41.614165+0000 mon.smithi087 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.195:0/2429332051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:43.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:43.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:44.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:44 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:43.254718+0000 mgr.smithi087.iydsqo (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:45.136 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 -- 172.21.15.195:0/406600884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 msgr2=0x7fea700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 --2- 172.21.15.195:0/406600884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fea58009fb0 tx=0x7fea58009b30).stop 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 -- 172.21.15.195:0/406600884 shutdown_connections 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 --2- 172.21.15.195:0/406600884 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 -- 172.21.15.195:0/406600884 >> 172.21.15.195:0/406600884 conn(0x7fea700f0b70 msgr2=0x7fea700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 -- 172.21.15.195:0/406600884 shutdown_connections 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.133+0000 7fea74f01700 1 -- 172.21.15.195:0/406600884 wait complete. 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea74f01700 1 Processor -- start 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea74f01700 1 -- start start 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea74f01700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:45.137 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea74f01700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700fb960 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea6e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea6e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:34082/0 (socket says 172.21.15.195:34082) 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea6e59c700 1 -- 172.21.15.195:0/2585570341 learned_addr learned my addr 172.21.15.195:0/2585570341 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.134+0000 7fea6e59c700 1 -- 172.21.15.195:0/2585570341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58005040 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea6e59c700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fea58005010 tx=0x7fea5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea58009320 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea58013380 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700fbb00 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea58014e50 con 0x7fea700f5410 2024-02-20T00:37:45.138 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.135+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700fbf90 con 0x7fea700f5410 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea58014420 con 0x7fea700f5410 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea70059100 con 0x7fea700f5410 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea66ffd700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 0x7fea5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea58040f40 con 0x7fea700f5410 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea6dd9b700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 0x7fea5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:45.139 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.136+0000 7fea6dd9b700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 0x7fea5c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fea600096a0 tx=0x7fea60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:45.142 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.140+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea580095f0 con 0x7fea700f5410 2024-02-20T00:37:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.344+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea700fafa0 con 0x7fea700f5410 2024-02-20T00:37:45.346 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.345+0000 7fea66ffd700 1 -- 172.21.15.195:0/2585570341 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea5801db70 con 0x7fea700f5410 2024-02-20T00:37:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:45.347 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:45.348 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 msgr2=0x7fea5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 0x7fea5c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fea600096a0 tx=0x7fea60006b40).stop 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 msgr2=0x7fea700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fea58005010 tx=0x7fea5800b040).stop 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 shutdown_connections 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fea5c02ff10 0x7fea5c0323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 --2- 172.21.15.195:0/2585570341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700f5410 0x7fea700fd4e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:45.349 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 >> 172.21.15.195:0/2585570341 conn(0x7fea700f0b70 msgr2=0x7fea7005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:45.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 shutdown_connections 2024-02-20T00:37:45.350 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:45.346+0000 7fea74f01700 1 -- 172.21.15.195:0/2585570341 wait complete. 2024-02-20T00:37:45.350 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:46.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:45.255014+0000 mgr.smithi087.iydsqo (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:46.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:46 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:45.345982+0000 mon.smithi087 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.195:0/2585570341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:47.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:47.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:48.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:48 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:47.255305+0000 mgr.smithi087.iydsqo (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.903+0000 7fdd27fff700 1 -- 172.21.15.195:0/4158999932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 msgr2=0x7fdd280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 --2- 172.21.15.195:0/4158999932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd280f66c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fdd18009fb0 tx=0x7fdd18009b30).stop 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 -- 172.21.15.195:0/4158999932 shutdown_connections 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 --2- 172.21.15.195:0/4158999932 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd280f66c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 -- 172.21.15.195:0/4158999932 >> 172.21.15.195:0/4158999932 conn(0x7fdd280f1ac0 msgr2=0x7fdd280f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 -- 172.21.15.195:0/4158999932 shutdown_connections 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 -- 172.21.15.195:0/4158999932 wait complete. 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 Processor -- start 2024-02-20T00:37:48.906 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 -- start start 2024-02-20T00:37:48.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.904+0000 7fdd27fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:48.907 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd27fff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28106ab0 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd26ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd26ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60736/0 (socket says 172.21.15.195:60736) 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd26ffd700 1 -- 172.21.15.195:0/2923080272 learned_addr learned my addr 172.21.15.195:0/2923080272 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd26ffd700 1 -- 172.21.15.195:0/2923080272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18005040 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.905+0000 7fdd26ffd700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fdd180089f0 tx=0x7fdd1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd18009320 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28106cb0 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd18013380 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd18014e20 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28107140 con 0x7fdd280f62a0 2024-02-20T00:37:48.908 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.906+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd180263a0 con 0x7fdd280f62a0 2024-02-20T00:37:48.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.907+0000 7fdd0f7fe700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:48.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.907+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd18040f80 con 0x7fdd280f62a0 2024-02-20T00:37:48.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.907+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28045ad0 con 0x7fdd280f62a0 2024-02-20T00:37:48.909 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.907+0000 7fdd267fc700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:48.910 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.907+0000 7fdd267fc700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 0x7fdd100323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:48.913 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:48.912+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd18012920 con 0x7fdd280f62a0 2024-02-20T00:37:49.119 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.117+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd28107e60 con 0x7fdd280f62a0 2024-02-20T00:37:49.120 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.118+0000 7fdd0f7fe700 1 -- 172.21.15.195:0/2923080272 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd1802b020 con 0x7fdd280f62a0 2024-02-20T00:37:49.120 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:49.120 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 msgr2=0x7fdd100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 0x7fdd100323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).stop 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 msgr2=0x7fdd281065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fdd180089f0 tx=0x7fdd1800b040).stop 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 shutdown_connections 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 --2- 172.21.15.195:0/2923080272 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdd280f62a0 0x7fdd281065c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 >> 172.21.15.195:0/2923080272 conn(0x7fdd280f1ac0 msgr2=0x7fdd280f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 shutdown_connections 2024-02-20T00:37:49.122 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:49.120+0000 7fdd27fff700 1 -- 172.21.15.195:0/2923080272 wait complete. 2024-02-20T00:37:49.123 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:49.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:49 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:49.119269+0000 mon.smithi087 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.195:0/2923080272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:50.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:50.649 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:50.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:50 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:49.255671+0000 mgr.smithi087.iydsqo (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 -- 172.21.15.195:0/2531059099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 msgr2=0x7fedd80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 --2- 172.21.15.195:0/2531059099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd80f8f20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fedc8009fb0 tx=0x7fedc8009b30).stop 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 -- 172.21.15.195:0/2531059099 shutdown_connections 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 --2- 172.21.15.195:0/2531059099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd80f8f20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 -- 172.21.15.195:0/2531059099 >> 172.21.15.195:0/2531059099 conn(0x7fedd80f09a0 msgr2=0x7fedd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.502+0000 7feddce9e700 1 -- 172.21.15.195:0/2531059099 shutdown_connections 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.503+0000 7feddce9e700 1 -- 172.21.15.195:0/2531059099 wait complete. 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.503+0000 7feddce9e700 1 Processor -- start 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.503+0000 7feddce9e700 1 -- start start 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.503+0000 7feddce9e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.503+0000 7feddce9e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8106af0 con 0x7fedd80601e0 2024-02-20T00:37:52.514 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedd77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedd77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:60762/0 (socket says 172.21.15.195:60762) 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedd77fe700 1 -- 172.21.15.195:0/1037996662 learned_addr learned my addr 172.21.15.195:0/1037996662 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedd77fe700 1 -- 172.21.15.195:0/1037996662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8005040 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedd77fe700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fedc8005010 tx=0x7fedc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc8009320 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8106cf0 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedc8013380 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc8014e50 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.504+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8107180 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.505+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedc80095f0 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.505+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8045ad0 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.505+0000 7fedbffff700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.506+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedc8040f70 con 0x7fedd80601e0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.506+0000 7fedd6ffd700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:52.515 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.506+0000 7fedd6ffd700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 0x7fedc00323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:52.516 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.510+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedc8014420 con 0x7fedd80601e0 2024-02-20T00:37:52.718 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.717+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedd8107fd0 con 0x7fedd80601e0 2024-02-20T00:37:52.719 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.718+0000 7fedbffff700 1 -- 172.21.15.195:0/1037996662 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedc8010230 con 0x7fedd80601e0 2024-02-20T00:37:52.719 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:52.720 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 msgr2=0x7fedc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 0x7fedc00323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fedcc0096a0 tx=0x7fedcc006b40).stop 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 msgr2=0x7fedd8106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fedc8005010 tx=0x7fedc800b040).stop 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 shutdown_connections 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 --2- 172.21.15.195:0/1037996662 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fedd80601e0 0x7fedd8106600 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 >> 172.21.15.195:0/1037996662 conn(0x7fedd80f09a0 msgr2=0x7fedd80f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 shutdown_connections 2024-02-20T00:37:52.721 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:52.719+0000 7feddce9e700 1 -- 172.21.15.195:0/1037996662 wait complete. 2024-02-20T00:37:52.722 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:52.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:52 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:51.256112+0000 mgr.smithi087.iydsqo (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:53.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:53 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:52.718997+0000 mon.smithi087 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.195:0/1037996662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:54.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:54.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:54.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:54 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:53.256409+0000 mgr.smithi087.iydsqo (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 -- 172.21.15.195:0/2032571196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 msgr2=0x7fc0980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 --2- 172.21.15.195:0/2032571196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0980f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fc08c009fb0 tx=0x7fc08c009b30).stop 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 -- 172.21.15.195:0/2032571196 shutdown_connections 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 --2- 172.21.15.195:0/2032571196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0980f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 -- 172.21.15.195:0/2032571196 >> 172.21.15.195:0/2032571196 conn(0x7fc0980f0b70 msgr2=0x7fc0980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 -- 172.21.15.195:0/2032571196 shutdown_connections 2024-02-20T00:37:56.459 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.457+0000 7fc09d84b700 1 -- 172.21.15.195:0/2032571196 wait complete. 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc09d84b700 1 Processor -- start 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc09d84b700 1 -- start start 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc09d84b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc09d84b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc098105bd0 con 0x7fc0980f5410 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc096ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:56.461 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc096ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:42208/0 (socket says 172.21.15.195:42208) 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.458+0000 7fc096ffd700 1 -- 172.21.15.195:0/4041286792 learned_addr learned my addr 172.21.15.195:0/4041286792 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc096ffd700 1 -- 172.21.15.195:0/4041286792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c005040 con 0x7fc0980f5410 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc096ffd700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fc08c005010 tx=0x7fc08c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c009320 con 0x7fc0980f5410 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc098105dd0 con 0x7fc0980f5410 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc08c013380 con 0x7fc0980f5410 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c014e50 con 0x7fc0980f5410 2024-02-20T00:37:56.462 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.459+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc098106260 con 0x7fc0980f5410 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.460+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc08c0095f0 con 0x7fc0980f5410 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.460+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc098045ad0 con 0x7fc0980f5410 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.460+0000 7fc07f7fe700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 0x7fc0800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.460+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc08c040e90 con 0x7fc0980f5410 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.460+0000 7fc0967fc700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 0x7fc0800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:37:56.463 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.461+0000 7fc0967fc700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 0x7fc0800323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc0880096a0 tx=0x7fc088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:37:56.466 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.465+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc08c014420 con 0x7fc0980f5410 2024-02-20T00:37:56.674 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.672+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc098107110 con 0x7fc0980f5410 2024-02-20T00:37:56.674 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.673+0000 7fc07f7fe700 1 -- 172.21.15.195:0/4041286792 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc08c010230 con 0x7fc0980f5410 2024-02-20T00:37:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:37:56.675 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:37:56.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 msgr2=0x7fc0800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:56.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 0x7fc0800323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc0880096a0 tx=0x7fc088006b40).stop 2024-02-20T00:37:56.676 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 msgr2=0x7fc0981056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fc08c005010 tx=0x7fc08c00b040).stop 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 shutdown_connections 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7fc08002ff10 0x7fc0800323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 --2- 172.21.15.195:0/4041286792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc0980f5410 0x7fc0981056e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 >> 172.21.15.195:0/4041286792 conn(0x7fc0980f0b70 msgr2=0x7fc0980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 shutdown_connections 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:37:56.675+0000 7fc09d84b700 1 -- 172.21.15.195:0/4041286792 wait complete. 2024-02-20T00:37:56.677 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:37:56.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:56 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:55.256727+0000 mgr.smithi087.iydsqo (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:37:57.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:56.674352+0000 mon.smithi087 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.195:0/4041286792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:37:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:56.676879+0000 mon.smithi087 (mon.0) 340 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:37:57.782 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:57 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:37:56.677095+0000 mon.smithi087 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.87:0/2804505963' entity='mgr.smithi087.iydsqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi087.iydsqo/trash_purge_schedule"}]: dispatch 2024-02-20T00:37:58.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:37:58.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6575da04-cf86-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:37:58.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:37:58 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:57.257038+0000 mgr.smithi087.iydsqo (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 -- 172.21.15.195:0/3842862487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 msgr2=0x7f54880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 --2- 172.21.15.195:0/3842862487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f54880f90f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f547c009fb0 tx=0x7f547c009b30).stop 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 -- 172.21.15.195:0/3842862487 shutdown_connections 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 --2- 172.21.15.195:0/3842862487 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f54880f90f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 -- 172.21.15.195:0/3842862487 >> 172.21.15.195:0/3842862487 conn(0x7f54880f0b70 msgr2=0x7f54880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 -- 172.21.15.195:0/3842862487 shutdown_connections 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.200+0000 7f548c90f700 1 -- 172.21.15.195:0/3842862487 wait complete. 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548c90f700 1 Processor -- start 2024-02-20T00:38:00.204 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548c90f700 1 -- start start 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548c90f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548c90f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5488106c70 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.195:42218/0 (socket says 172.21.15.195:42218) 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.201+0000 7f548659c700 1 -- 172.21.15.195:0/2348076758 learned_addr learned my addr 172.21.15.195:0/2348076758 (peer_addr_for_me v2:172.21.15.195:0/0) 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f548659c700 1 -- 172.21.15.195:0/2348076758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c005040 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f548659c700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f547c0089f0 tx=0x7f547c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f547c009320 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5488106e70 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f547c013380 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f547c014e20 con 0x7f5488060280 2024-02-20T00:38:00.205 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.202+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5488107300 con 0x7f5488060280 2024-02-20T00:38:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f547c0095f0 con 0x7f5488060280 2024-02-20T00:38:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5488045ad0 con 0x7f5488060280 2024-02-20T00:38:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f5476ffd700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 0x7f5470032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:38:00.206 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f547c040f00 con 0x7f5488060280 2024-02-20T00:38:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f5485d9b700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 0x7f5470032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:38:00.207 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.203+0000 7f5485d9b700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 0x7f5470032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f54780096a0 tx=0x7f5478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:38:00.209 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.208+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f547c014420 con 0x7f5488060280 2024-02-20T00:38:00.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.414+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54881081b0 con 0x7f5488060280 2024-02-20T00:38:00.416 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.415+0000 7f5476ffd700 1 -- 172.21.15.195:0/2348076758 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f547c01d980 con 0x7f5488060280 2024-02-20T00:38:00.417 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-02-20T00:38:00.417 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"6575da04-cf86-11ee-95bb-87774f69a715","modified":"2024-02-20T00:24:49.938605Z","created":"2024-02-20T00:24:49.938605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi087","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:38:00.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 msgr2=0x7f5470032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:38:00.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 0x7f5470032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f54780096a0 tx=0x7f5478006b40).stop 2024-02-20T00:38:00.418 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 msgr2=0x7f5488106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f547c0089f0 tx=0x7f547c00b040).stop 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 shutdown_connections 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:6800/3526433496,v1:172.21.15.87:6801/3526433496] conn(0x7f547002ff60 0x7f5470032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 --2- 172.21.15.195:0/2348076758 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5488060280 0x7f5488106780 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.416+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 >> 172.21.15.195:0/2348076758 conn(0x7f54880f0b70 msgr2=0x7f54880f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.417+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 shutdown_connections 2024-02-20T00:38:00.419 INFO:teuthology.orchestra.run.smithi195.stderr:2024-02-20T00:38:00.417+0000 7f548c90f700 1 -- 172.21.15.195:0/2348076758 wait complete. 2024-02-20T00:38:00.420 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-02-20T00:38:00.781 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:00 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: cluster 2024-02-20T00:37:59.257357+0000 mgr.smithi087.iydsqo (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:38:01.177 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-20T00:38:01.179 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T00:38:01.179 DEBUG:teuthology.orchestra.run.smithi087:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:38:01.203 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:38:01.222 INFO:tasks.cephadm:Stopping all daemons... 2024-02-20T00:38:01.222 INFO:tasks.cephadm.mon.smithi087:Stopping mon.smithi087... 2024-02-20T00:38:01.222 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087 2024-02-20T00:38:01.531 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:01 smithi087 systemd[1]: Stopping Ceph mon.smithi087 for 6575da04-cf86-11ee-95bb-87774f69a715... 2024-02-20T00:38:01.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:01 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: audit 2024-02-20T00:38:00.416050+0000 mon.smithi087 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.195:0/2348076758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:38:01.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:01 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:38:01.426+0000 7ff4c6d71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi087 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T00:38:01.532 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:01 smithi087 ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087[38074]: debug 2024-02-20T00:38:01.426+0000 7ff4c6d71700 -1 mon.smithi087@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T00:38:02.032 INFO:journalctl@ceph.mon.smithi087.smithi087.stdout:Feb 20 00:38:01 smithi087 podman[44216]: ceph-6575da04-cf86-11ee-95bb-87774f69a715-mon.smithi087 2024-02-20T00:38:02.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi087.service' 2024-02-20T00:38:02.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T00:38:02.277 INFO:tasks.cephadm.mon.smithi087:Stopped mon.smithi087 2024-02-20T00:38:02.277 INFO:tasks.cephadm.mon.smithi195:Stopping mon.smithi195... 2024-02-20T00:38:02.278 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195 2024-02-20T00:38:02.317 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195.service: Unit ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195.service not loaded. 2024-02-20T00:38:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-20T00:38:02.346 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi195" Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195' 2024-02-20T00:38:02.347 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T00:38:02.348 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/ubuntu@smithi087.front.sepia.ceph.com/crash 2024-02-20T00:38:02.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/crash -- . 2024-02-20T00:38:02.395 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/ubuntu@smithi195.front.sepia.ceph.com/crash 2024-02-20T00:38:02.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/crash -- . 2024-02-20T00:38:02.437 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/6575da04-cf86-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-20T00:38:02.437 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-02-20T00:38:02.438 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T00:38:02.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T00:38:02.473 INFO:tasks.cephadm:Compressing logs... 2024-02-20T00:38:02.473 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T00:38:02.515 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T00:38:02.541 INFO:teuthology.orchestra.run.smithi087.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T00:38:02.541 INFO:teuthology.orchestra.run.smithi087.stderr:: No such file or directory 2024-02-20T00:38:02.541 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mon.smithi087.log 2024-02-20T00:38:02.541 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T00:38:02.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-20T00:38:02.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mon.smithi087.log: /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-20T00:38:02.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mgr.smithi087.iydsqo.log 2024-02-20T00:38:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.log 2024-02-20T00:38:02.544 INFO:teuthology.orchestra.run.smithi087.stderr: 90.2% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T00:38:02.544 INFO:teuthology.orchestra.run.smithi087.stderr: 90.3% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-20T00:38:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mgr.smithi087.iydsqo.log: 88.9% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-20T00:38:02.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-20T00:38:02.546 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T00:38:02.546 INFO:teuthology.orchestra.run.smithi195.stderr:: No such file or directory 2024-02-20T00:38:02.547 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T00:38:02.547 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T00:38:02.548 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T00:38:02.549 INFO:teuthology.orchestra.run.smithi195.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T00:38:02.550 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-02-20T00:38:02.550 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.015s 2024-02-20T00:38:02.551 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m0.015s 2024-02-20T00:38:02.551 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.011s 2024-02-20T00:38:02.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip: /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mgr.smithi087.iydsqo.log: file size changed while zipping 2024-02-20T00:38:02.645 INFO:teuthology.orchestra.run.smithi087.stderr: 92.7% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mgr.smithi087.iydsqo.log.gz 2024-02-20T00:38:02.718 INFO:teuthology.orchestra.run.smithi087.stderr: 90.6% -- replaced with /var/log/ceph/6575da04-cf86-11ee-95bb-87774f69a715/ceph-mon.smithi087.log.gz 2024-02-20T00:38:02.719 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-20T00:38:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m0.189s 2024-02-20T00:38:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m0.288s 2024-02-20T00:38:02.719 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.023s 2024-02-20T00:38:02.720 INFO:tasks.cephadm:Archiving logs... 2024-02-20T00:38:02.720 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/ubuntu@smithi087.front.sepia.ceph.com/log 2024-02-20T00:38:02.721 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T00:38:02.847 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/ubuntu@smithi195.front.sepia.ceph.com/log 2024-02-20T00:38:02.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T00:38:02.880 INFO:tasks.cephadm:Removing cluster... 2024-02-20T00:38:02.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6575da04-cf86-11ee-95bb-87774f69a715 --force 2024-02-20T00:41:04.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6575da04-cf86-11ee-95bb-87774f69a715 --force 2024-02-20T00:41:04.551 INFO:tasks.cephadm:Removing cephadm ... 2024-02-20T00:41:04.551 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T00:41:04.570 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T00:41:04.588 INFO:tasks.cephadm:Teardown complete 2024-02-20T00:41:04.588 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195' 2024-02-20T00:41:04.827 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4302da74f28a400381cbd006f29786f4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195' 2024-02-20T00:41:04.832 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T00:41:04.844 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T00:41:04.852 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T00:41:04.853 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T00:41:04.876 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T00:41:04.952 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T00:41:04.961 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T00:41:04.961 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:41:04.963 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:41:04.979 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2024-02-20T00:41:04.981 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +1194us[+1209us] +/- 111ms 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -873us[ -858us] +/- 64ms 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +238us[ +238us] +/- 71ms 2024-02-20T00:41:04.983 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:41:04.986 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:41:04.986 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-02-20T00:41:04.986 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +1253us[+1238us] +/- 111ms 2024-02-20T00:41:04.987 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -813us[ -828us] +/- 64ms 2024-02-20T00:41:04.987 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +286us[ +286us] +/- 71ms 2024-02-20T00:41:04.987 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:41:04.987 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T00:41:04.996 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T00:41:04.997 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T00:41:05.005 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T00:41:05.086 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T00:41:05.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:41:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:41:05.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:41:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:41:05.212 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T00:41:05.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T00:41:05.233 INFO:teuthology.task.internal:Duration was 1549.166968 seconds 2024-02-20T00:41:05.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T00:41:05.242 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T00:41:05.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T00:41:05.245 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T00:41:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:41:05.298 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:41:05.580 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-20T00:41:05.580 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-02-20T00:41:05.580 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T00:41:05.604 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-02-20T00:41:05.604 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T00:41:05.627 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-20T00:41:05.627 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T00:41:05.646 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T00:41:05.717 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-20T00:41:05.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T00:41:05.720 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T00:41:05.835 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T00:41:05.845 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T00:41:05.846 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T00:41:05.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T00:41:05.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T00:41:05.910 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:41:05.915 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:41:05.939 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-02-20T00:41:05.967 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-02-20T00:41:05.986 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:41:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:41:06.021 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:41:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:41:06.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T00:41:06.054 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T00:41:06.055 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/smithi087 2024-02-20T00:41:06.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T00:41:06.101 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154/remote/smithi195 2024-02-20T00:41:06.102 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T00:41:06.142 INFO:teuthology.task.internal:Removing archive directory... 2024-02-20T00:41:06.143 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T00:41:06.145 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T00:41:06.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T00:41:06.204 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T00:41:06.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T00:41:06.214 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T00:41:06.214 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T00:41:06.217 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T00:41:06.231 INFO:teuthology.orchestra.run.smithi087.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 00:41 /home/ubuntu/cephtest 2024-02-20T00:41:06.251 INFO:teuthology.orchestra.run.smithi195.stdout: 525694 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 00:41 /home/ubuntu/cephtest 2024-02-20T00:41:06.252 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T00:41:06.263 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T00:41:06.285 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T00:41:06.308 DEBUG:teuthology.nuke:shortname: smithi087 2024-02-20T00:41:06.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T00:41:06.332 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 00:07:10.550577', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ0W8H30qUQ4q7DBgNdqzwBG8unR5I58XXYMyRLBdfufnsdKVN1keaU5CaMPJRrj8msTB3Ic+b80e1pw2F70iHM='} 2024-02-20T00:41:06.338 DEBUG:teuthology.nuke:shortname: smithi195 2024-02-20T00:41:06.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T00:41:06.366 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568154', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 00:07:10.548640', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7d1fye6oSJY3TGwuzE76VGS02yhrXNR5+sAcvx8Uv1YiDmq/AzXKfmI+CvEyl5pFfKeeEClt8RljHOOWxfu94='} 2024-02-20T00:41:06.393 INFO:teuthology.orchestra.console.smithi087:Power off 2024-02-20T00:41:06.393 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T00:41:06.416 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2024-02-20T00:41:06.417 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:06.434 INFO:teuthology.orchestra.console.smithi195:Power off 2024-02-20T00:41:06.435 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T00:41:06.445 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-02-20T00:41:06.460 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-02-20T00:41:06.460 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:06.485 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-20T00:41:10.445 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:10.486 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:10.556 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-02-20T00:41:10.597 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-02-20T00:41:14.560 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:14.598 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:14.670 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-02-20T00:41:14.707 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-02-20T00:41:14.808 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-02-20T00:41:18.675 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:41:18.784 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-02-20T00:41:18.886 INFO:teuthology.orchestra.console.smithi087:Power off completed 2024-02-20T00:41:18.990 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1549.166967868805 failure_reason: 'Command failed on smithi195 with status 5: ''sudo systemctl stop ceph-6575da04-cf86-11ee-95bb-87774f69a715@mon.smithi195''' flavor: default owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4302da74f28a400381cbd006f29786f4 status: fail success: false 2024-02-20T00:41:18.991 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T00:41:19.074 INFO:teuthology.run:FAIL