2024-02-23T00:00:04.621 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-23T00:00:04.621 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395 --verbose --name yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/orig.config.yaml 2024-02-23T00:00:04.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-23T00:00:04.798 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395 branch: pacific-release description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7571395' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(CEPHADM_STRAY_DAEMON\) - \(FS_DEGRADED\) - FS_ - \(CEPHADM_ - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - acting - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding - is down - osds down sha1: eb66ed921e744301e6863be6353618d63967ea59 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: eb66ed921e744301e6863be6353618d63967ea59 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ksirivad-pacific-release-whitelist sha1: 4f758cd8e8854d1f126da86e254076852bf059a9 owner: scheduled_yuriw@teuthology priority: 61 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 264 sha1: eb66ed921e744301e6863be6353618d63967ea59 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ksirivad-pacific-release-whitelist suite_path: /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa suite_relpath: qa suite_repo: https://github.com/kamoltat/ceph.git suite_sha1: 4f758cd8e8854d1f126da86e254076852bf059a9 targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAw66Tluu/5EKzyB/erNNt8aJIbaZk+GzOE3LkTuiVuu3criI50enx46TASsXmmP3ZNqFnmQbRlfJqHlCr+Jfs= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpYTI8SAlR2H/0ngjM5LP1E9T4AWk2Q31Wy3Sdi4GjlZr53ndDadIrd5zUNflUMiaEgrf1uQxihOO+pcT4DKwE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-22_21:39:39 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-23T00:00:04.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa; will attempt to use it 2024-02-23T00:00:04.799 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks 2024-02-23T00:00:04.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-23T00:00:04.801 INFO:teuthology.task.internal:Checking packages... 2024-02-23T00:00:04.824 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'eb66ed921e744301e6863be6353618d63967ea59' 2024-02-23T00:00:04.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-23T00:00:04.824 INFO:teuthology.packaging:ref: None 2024-02-23T00:00:04.824 INFO:teuthology.packaging:tag: None 2024-02-23T00:00:04.824 INFO:teuthology.packaging:branch: pacific-release 2024-02-23T00:00:04.824 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:00:04.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-23T00:00:05.046 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-550.geb66ed92 2024-02-23T00:00:05.048 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-23T00:00:05.055 INFO:teuthology.task.internal:no buildpackages task found 2024-02-23T00:00:05.055 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-23T00:00:05.061 INFO:teuthology.task.internal:Saving configuration 2024-02-23T00:00:05.079 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-23T00:00:05.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-23T00:00:05.111 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:54:10.360843', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAw66Tluu/5EKzyB/erNNt8aJIbaZk+GzOE3LkTuiVuu3criI50enx46TASsXmmP3ZNqFnmQbRlfJqHlCr+Jfs='} 2024-02-23T00:00:05.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:54:10.362575', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpYTI8SAlR2H/0ngjM5LP1E9T4AWk2Q31Wy3Sdi4GjlZr53ndDadIrd5zUNflUMiaEgrf1uQxihOO+pcT4DKwE='} 2024-02-23T00:00:05.140 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-23T00:00:05.147 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-23T00:00:05.148 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-23T00:00:05.148 INFO:teuthology.run_tasks:Running task console_log... 2024-02-23T00:00:05.220 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fde2bcd30d0>, signals=[15]) 2024-02-23T00:00:05.221 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-23T00:00:05.229 INFO:teuthology.task.internal:Opening connections... 2024-02-23T00:00:05.230 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-02-23T00:00:05.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:00:05.332 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-02-23T00:00:05.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:00:05.447 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-23T00:00:05.453 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-02-23T00:00:05.475 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-02-23T00:00:05.475 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-02-23T00:00:05.535 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-02-23T00:00:05.536 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-23T00:00:05.537 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-23T00:00:05.537 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-23T00:00:05.537 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-02-23T00:00:05.581 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-02-23T00:00:05.603 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-02-23T00:00:05.603 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-23T00:00:05.665 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-02-23T00:00:05.666 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-23T00:00:05.666 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-23T00:00:05.666 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-23T00:00:05.666 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-02-23T00:00:05.706 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-23T00:00:05.718 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-23T00:00:05.724 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-23T00:00:05.724 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-02-23T00:00:05.726 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-02-23T00:00:05.747 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-23T00:00:05.754 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-23T00:00:05.754 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-02-23T00:00:05.787 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-02-23T00:00:05.814 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-23T00:00:05.867 INFO:teuthology.run_tasks:Running task kernel... 2024-02-23T00:00:05.881 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-23T00:00:05.881 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-23T00:00:05.882 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-23T00:00:05.882 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-23T00:00:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:00:05.903 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-23T00:00:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:00:05.904 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-02-23T00:00:05.964 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-489.el8.x86_64 2024-02-23T00:00:05.964 INFO:teuthology.task.kernel:Running kernel on smithi090: 4.18.0-489.el8.x86_64 2024-02-23T00:00:05.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-02-23T00:00:05.965 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-489.el8.x86_64 2024-02-23T00:00:05.966 INFO:teuthology.task.kernel:Running kernel on smithi118: 4.18.0-489.el8.x86_64 2024-02-23T00:00:05.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-02-23T00:00:21.940 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - AppStream 2.2 MB/s | 27 MB 00:12 2024-02-23T00:00:27.584 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-02-23T00:00:27.832 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - AppStream 1.4 MB/s | 27 MB 00:19 2024-02-23T00:00:29.368 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras 98 kB/s | 18 kB 00:00 2024-02-23T00:00:29.818 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-02-23T00:00:30.045 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-23T00:00:31.480 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-02-23T00:00:33.665 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-23T00:00:35.280 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 481 kB/s | 24 kB 00:00 2024-02-23T00:00:35.590 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras 52 kB/s | 18 kB 00:00 2024-02-23T00:00:36.067 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-23T00:00:36.326 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-02-23T00:00:37.712 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-02-23T00:00:38.252 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-23T00:00:38.253 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-23T00:00:38.370 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:00:38.370 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:00:38.370 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 4.18.0-540.el8 baseos 10 M 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 4.18.0-540.el8 baseos 44 M 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 4.18.0-540.el8 baseos 36 M 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout:Install 3 Packages 2024-02-23T00:00:38.371 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:00:38.372 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 90 M 2024-02-23T00:00:38.372 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 97 M 2024-02-23T00:00:38.372 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:00:39.034 INFO:teuthology.orchestra.run.smithi090.stdout:(1/3): kernel-4.18.0-540.el8.x86_64.rpm 17 MB/s | 10 MB 00:00 2024-02-23T00:00:40.142 INFO:teuthology.orchestra.run.smithi090.stdout:(2/3): kernel-modules-4.18.0-540.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2024-02-23T00:00:40.726 INFO:teuthology.orchestra.run.smithi090.stdout:(3/3): kernel-core-4.18.0-540.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-02-23T00:00:40.726 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:00:40.726 INFO:teuthology.orchestra.run.smithi090.stdout:Total 38 MB/s | 90 MB 00:02 2024-02-23T00:00:41.410 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:00:41.518 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 322 kB/s | 24 kB 00:00 2024-02-23T00:00:41.844 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:00:41.844 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:00:43.140 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:00:43.186 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:00:44.590 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-23T00:00:44.590 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-23T00:00:44.709 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 4.18.0-540.el8 baseos 10 M 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 4.18.0-540.el8 baseos 44 M 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 4.18.0-540.el8 baseos 36 M 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:00:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:00:44.711 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:00:44.711 INFO:teuthology.orchestra.run.smithi118.stdout:Install 3 Packages 2024-02-23T00:00:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:00:44.711 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 90 M 2024-02-23T00:00:44.711 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 97 M 2024-02-23T00:00:44.712 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:00:45.886 INFO:teuthology.orchestra.run.smithi118.stdout:(1/3): kernel-4.18.0-540.el8.x86_64.rpm 9.7 MB/s | 10 MB 00:01 2024-02-23T00:00:45.968 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:00:46.483 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:00:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:00:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:00:53.436 INFO:teuthology.orchestra.run.smithi118.stdout:(2/3): kernel-modules-4.18.0-540.el8.x86_64.rpm 4.2 MB/s | 36 MB 00:08 2024-02-23T00:00:53.966 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:00:54.257 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:55.261 INFO:teuthology.orchestra.run.smithi118.stdout:(3/3): kernel-core-4.18.0-540.el8.x86_64.rpm 4.2 MB/s | 44 MB 00:10 2024-02-23T00:00:55.261 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:00:55.261 INFO:teuthology.orchestra.run.smithi118.stdout:Total 8.6 MB/s | 90 MB 00:10 2024-02-23T00:00:55.944 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:00:56.367 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:00:56.368 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:00:57.658 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:00:57.696 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:01:00.478 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:01:00.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:01:04.576 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:01:04.993 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:01:08.434 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:01:08.716 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:16.662 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:16.685 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:17.028 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:17.029 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:01:17.029 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:01:17.526 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:17.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:01:17.527 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:01:17.527 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-4.18.0-540.el8.x86_64 kernel-core-4.18.0-540.el8.x86_64 2024-02-23T00:01:17.527 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-4.18.0-540.el8.x86_64 2024-02-23T00:01:17.527 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:01:17.527 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:01:17.674 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:01:17.892 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:01:17.892 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:01:17.892 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-540.el8.x86_64 2024-02-23T00:01:17.892 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-23T00:01:17.892 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-23T00:01:17.892 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-23T00:01:17.893 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-02-23T00:01:17.893 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:01:18.102 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:01:18.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:01:18.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.02 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 129.el8 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Group : System Environment/Base 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 9265586 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-23T00:01:18.136 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Relocations : (not relocatable) 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : CentOS BuildSystem 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-23T00:01:18.137 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-02-23T00:01:18.138 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:01:18.138 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-02-23T00:01:18.138 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-23T00:01:18.139 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 4.18.0-540.el8.x86_64 2024-02-23T00:01:18.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-23T00:01:20.047 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-02-23T00:01:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-02-23T00:01:22.529 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-02-23T00:01:22.592 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.WbDnJozGbb 2024-02-23T00:01:22.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WbDnJozGbb 2024-02-23T00:01:22.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.WbDnJozGbb 2024-02-23T00:01:22.774 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.WbDnJozGbb is 6KB 2024-02-23T00:01:22.822 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.WbDnJozGbb 2024-02-23T00:01:22.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-02-23T00:01:22.908 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-23T00:01:22.909 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-23T00:01:22.909 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-23T00:01:22.909 INFO:teuthology.orchestra.run.smithi090.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64.conf 2024-02-23T00:01:22.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64 2024-02-23T00:01:23.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-02-23T00:01:30.723 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:30.748 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:31.103 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:31.103 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:01:31.103 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:01:32.091 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:01:32.091 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:01:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:01:32.092 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-4.18.0-540.el8.x86_64 kernel-core-4.18.0-540.el8.x86_64 2024-02-23T00:01:32.092 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-4.18.0-540.el8.x86_64 2024-02-23T00:01:32.092 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:01:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:01:32.299 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:01:32.520 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:01:32.521 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:01:32.521 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-540.el8.x86_64 2024-02-23T00:01:32.521 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-23T00:01:32.521 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-23T00:01:32.521 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-23T00:01:32.521 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-02-23T00:01:32.521 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:01:32.730 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:01:32.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:01:32.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-02-23T00:01:32.763 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-02-23T00:01:32.763 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.02 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 129.el8 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Group : System Environment/Base 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 9265586 2024-02-23T00:01:32.764 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-02-23T00:01:32.765 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-23T00:01:32.765 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-23T00:01:32.765 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-23T00:01:32.765 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-23T00:01:32.765 INFO:teuthology.orchestra.run.smithi118.stdout:Relocations : (not relocatable) 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : CentOS BuildSystem 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-23T00:01:32.766 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-02-23T00:01:32.767 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:01:32.767 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-02-23T00:01:32.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-23T00:01:32.768 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 4.18.0-540.el8.x86_64 2024-02-23T00:01:32.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-23T00:01:34.674 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-02-23T00:01:36.557 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-02-23T00:01:36.559 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-02-23T00:01:36.581 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.me1wFIZhBD 2024-02-23T00:01:36.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.me1wFIZhBD 2024-02-23T00:01:36.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.me1wFIZhBD 2024-02-23T00:01:36.833 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.me1wFIZhBD is 6KB 2024-02-23T00:01:36.882 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.me1wFIZhBD 2024-02-23T00:01:36.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-02-23T00:01:36.966 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-23T00:01:36.966 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-23T00:01:36.966 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-23T00:01:36.966 INFO:teuthology.orchestra.run.smithi118.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64.conf 2024-02-23T00:01:36.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64 2024-02-23T00:01:37.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-02-23T00:01:53.094 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-23T00:01:53.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:01:53.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:07.126 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-23T00:02:07.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:02:07.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:11.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-23T00:02:20.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:02:20.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:23.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-23T00:02:25.472 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-02-23T00:02:34.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:02:34.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:35.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:02:35.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:37.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-02-23T00:02:38.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-23T00:02:49.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:02:49.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:52.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-02-23T00:02:53.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:02:53.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:03:07.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:03:07.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:03:12.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-02-23T00:03:30.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:03:30.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:03:31.386 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-23T00:03:32.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:03:32.069 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-540.el8.x86_64"... 2024-02-23T00:03:32.070 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-02-23T00:03:32.086 INFO:teuthology.orchestra.run.smithi090.stdout:4.18.0-540.el8.x86_64 2024-02-23T00:03:32.086 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-540.el8.x86_64 vs 4.18.0-540.el8.x86_64 2024-02-23T00:03:32.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-23T00:03:32.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-23T00:03:33.087 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-23T00:03:33.088 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-23T00:03:33.202 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-02-23T00:03:33.221 DEBUG:teuthology.parallel:result is None 2024-02-23T00:03:39.560 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-02-23T00:03:39.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:03:39.811 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-540.el8.x86_64"... 2024-02-23T00:03:39.811 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-02-23T00:03:39.828 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-540.el8.x86_64 2024-02-23T00:03:39.828 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-540.el8.x86_64 vs 4.18.0-540.el8.x86_64 2024-02-23T00:03:39.828 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-23T00:03:39.828 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-23T00:03:40.829 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-23T00:03:40.830 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-23T00:03:40.918 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-02-23T00:03:40.938 DEBUG:teuthology.parallel:result is None 2024-02-23T00:03:40.938 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-23T00:03:40.951 INFO:teuthology.task.internal:Creating test directory... 2024-02-23T00:03:40.951 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-23T00:03:40.954 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-23T00:03:40.975 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-23T00:03:40.985 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-23T00:03:40.993 INFO:teuthology.task.internal:Creating archive directory... 2024-02-23T00:03:40.993 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-23T00:03:41.015 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-23T00:03:41.062 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-23T00:03:41.074 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-23T00:03:41.075 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-23T00:03:41.094 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-23T00:03:41.133 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:03:41.134 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:03:41.144 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:03:41.145 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:03:41.147 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-23T00:03:41.178 INFO:teuthology.task.internal:Configuring sudo... 2024-02-23T00:03:41.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-23T00:03:41.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-23T00:03:41.218 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-23T00:03:41.234 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-23T00:03:41.234 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-23T00:03:41.258 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-23T00:03:41.278 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:03:41.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:03:41.410 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:03:41.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:03:41.537 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:03:41.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-23T00:03:41.604 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:03:41.660 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:03:41.715 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:03:41.775 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:03:41.811 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:03:41.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-23T00:03:41.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-02-23T00:03:41.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-02-23T00:03:41.947 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:03:41.994 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:03:42.466 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-23T00:03:42.476 INFO:teuthology.task.internal:Starting timer... 2024-02-23T00:03:42.476 INFO:teuthology.run_tasks:Running task pcp... 2024-02-23T00:03:42.489 INFO:teuthology.run_tasks:Running task selinux... 2024-02-23T00:03:42.498 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-23T00:03:42.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-02-23T00:03:42.566 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-02-23T00:03:42.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-02-23T00:03:42.676 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-02-23T00:03:42.677 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-23T00:03:42.677 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-02-23T00:03:42.701 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-02-23T00:03:42.701 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-02-23T00:03:42.722 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-02-23T00:03:42.723 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-02-23T00:03:42.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:03:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:03:42.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:03:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:03:42.798 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-23T00:03:42.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-02-23T00:03:42.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-02-23T00:03:42.872 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-23T00:03:42.891 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-23T00:03:42.909 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-23T00:03:42.925 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-23T00:13:54.045 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-02-23T00:13:54.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:13:54.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:13:54.148 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-02-23T00:13:54.216 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-02-23T00:13:54.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:13:54.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:13:54.329 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-02-23T00:13:54.403 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-02-23T00:13:54.404 INFO:teuthology.run_tasks:Running task clock... 2024-02-23T00:13:54.422 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-23T00:13:54.422 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-23T00:13:54.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:13:54.426 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-23T00:13:54.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:13:54.457 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-23T00:13:54.472 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-23T00:13:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-23T00:13:54.490 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-23T00:13:54.535 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-02-23T00:13:54.546 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-02-23T00:13:54.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-02-23T00:13:54.559 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-23T00:13:54.567 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-02-23T00:13:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-23T00:13:54.580 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-23T00:13:54.592 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-23T00:13:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.671 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:54.676 INFO:teuthology.run_tasks:Running task pexec... 2024-02-23T00:13:54.686 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-23T00:13:54.687 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-02-23T00:13:54.687 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:13:54.688 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi118.front.sepia.ceph.com 2024-02-23T00:13:54.688 DEBUG:teuthology.orchestra.run.smithi118:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:13:55.471 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:23 ago on Fri 23 Feb 2024 12:12:32 AM UTC. 2024-02-23T00:13:55.497 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:24 ago on Fri 23 Feb 2024 12:12:31 AM UTC. 2024-02-23T00:13:57.484 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:13:57.485 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout:Resetting modules: 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:13:57.486 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:13:57.554 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:13:57.555 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:13:57.555 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:13:57.555 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:13:57.555 INFO:teuthology.orchestra.run.smithi118.stdout:Resetting modules: 2024-02-23T00:13:57.556 INFO:teuthology.orchestra.run.smithi118.stdout: container-tools 2024-02-23T00:13:57.556 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:13:57.556 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:13:57.556 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:13:57.556 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:13:57.730 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:13:57.835 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:13:58.597 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:26 ago on Fri 23 Feb 2024 12:12:32 AM UTC. 2024-02-23T00:13:58.677 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:27 ago on Fri 23 Feb 2024 12:12:31 AM UTC. 2024-02-23T00:14:00.631 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:14:00.664 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:14:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:14:00.664 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:14:00.664 INFO:teuthology.orchestra.run.smithi090.stdout:Installing group/module packages: 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: crun x86_64 1.14.1-1.module_el8+951+32019cde appstream 257 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-23T00:14:00.665 INFO:teuthology.orchestra.run.smithi090.stdout:Installing module profiles: 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools/common 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling module streams: 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: container-tools rhel8 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-02-23T00:14:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:14:00.680 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 27 M 2024-02-23T00:14:00.680 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 91 M 2024-02-23T00:14:00.681 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:14:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:14:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:14:00.763 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:14:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:14:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:Installing group/module packages: 2024-02-23T00:14:00.763 INFO:teuthology.orchestra.run.smithi118.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: crun x86_64 1.14.1-1.module_el8+951+32019cde appstream 257 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-23T00:14:00.764 INFO:teuthology.orchestra.run.smithi118.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Installing module profiles: 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: container-tools/common 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling module streams: 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: container-tools rhel8 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-02-23T00:14:00.765 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:14:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 27 M 2024-02-23T00:14:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 91 M 2024-02-23T00:14:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:14:01.064 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 3.2 MB/s | 688 kB 00:00 2024-02-23T00:14:01.123 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.7 MB/s | 1.0 MB 00:00 2024-02-23T00:14:01.139 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 502 kB/s | 38 kB 00:00 2024-02-23T00:14:01.157 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 3.4 MB/s | 688 kB 00:00 2024-02-23T00:14:01.192 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): crun-1.14.1-1.module_el8+951+32019cde.x 3.8 MB/s | 257 kB 00:00 2024-02-23T00:14:01.193 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 4.3 MB/s | 1.0 MB 00:00 2024-02-23T00:14:01.216 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 645 kB/s | 38 kB 00:00 2024-02-23T00:14:01.240 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.5 MB/s | 180 kB 00:00 2024-02-23T00:14:01.245 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): crun-1.14.1-1.module_el8+951+32019cde.x 5.0 MB/s | 257 kB 00:00 2024-02-23T00:14:01.284 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 592 kB/s | 25 kB 00:00 2024-02-23T00:14:01.292 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.5 MB/s | 180 kB 00:00 2024-02-23T00:14:01.324 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 2.2 MB/s | 94 kB 00:00 2024-02-23T00:14:01.339 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 600 kB/s | 25 kB 00:00 2024-02-23T00:14:01.375 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 2.2 MB/s | 94 kB 00:00 2024-02-23T00:14:01.651 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.4 MB/s | 4.1 MB 00:00 2024-02-23T00:14:01.657 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 7.9 MB/s | 4.1 MB 00:00 2024-02-23T00:14:01.791 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 12 MB/s | 9.7 MB 00:00 2024-02-23T00:14:01.851 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 871 kB/s | 53 kB 00:00 2024-02-23T00:14:01.916 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 9.1 MB/s | 9.7 MB 00:01 2024-02-23T00:14:01.970 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): skopeo-1.14.3-0.1.module_el8+885+7da14 15 MB/s | 8.8 MB 00:00 2024-02-23T00:14:01.975 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 8.0 MB/s | 2.5 MB 00:00 2024-02-23T00:14:01.994 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 288 kB/s | 41 kB 00:00 2024-02-23T00:14:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 630 kB/s | 53 kB 00:00 2024-02-23T00:14:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 818 kB/s | 41 kB 00:00 2024-02-23T00:14:02.036 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): toolbox-0.0.99.5-1.module_el8+847+7863 6.6 MB/s | 2.5 MB 00:00 2024-02-23T00:14:02.036 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:14:02.036 INFO:teuthology.orchestra.run.smithi118.stdout:Total 22 MB/s | 27 MB 00:01 2024-02-23T00:14:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 11 MB/s | 8.8 MB 00:00 2024-02-23T00:14:02.143 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:14:02.143 INFO:teuthology.orchestra.run.smithi090.stdout:Total 19 MB/s | 27 MB 00:01 2024-02-23T00:14:02.303 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:14:02.324 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:14:02.324 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:14:02.410 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:14:02.431 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:14:02.431 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:14:02.606 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:14:02.609 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:14:02.733 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:14:02.736 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:14:03.306 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:14:03.440 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:14:04.281 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-23T00:14:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-23T00:14:04.498 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:14:04.572 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:14:04.669 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-23T00:14:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-23T00:14:04.894 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-23T00:14:04.979 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-23T00:14:05.143 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-23T00:14:05.204 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-23T00:14:05.704 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-23T00:14:05.771 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-23T00:14:06.708 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-02-23T00:14:06.802 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-02-23T00:14:06.954 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-23T00:14:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-23T00:14:07.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : crun-1.14.1-1.module_el8+951+32019cde.x86_64 9/13 2024-02-23T00:14:07.285 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : crun-1.14.1-1.module_el8+951+32019cde.x86_64 9/13 2024-02-23T00:14:07.651 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-23T00:14:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-23T00:14:07.871 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:14:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:14:08.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:14:08.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:14:08.368 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:14:08.392 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:14:08.787 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:14:08.787 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : crun-1.14.1-1.module_el8+951+32019cde.x86_64 5/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:14:08.788 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:14:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:14:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-23T00:14:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:14:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : crun-1.14.1-1.module_el8+951+32019cde.x86_64 5/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:14:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: crun-1.14.1-1.module_el8+951+32019cde.x86_64 2024-02-23T00:14:09.355 INFO:teuthology.orchestra.run.smithi118.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:14:09.356 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:14:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: crun-1.14.1-1.module_el8+951+32019cde.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:14:09.360 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:14:09.628 DEBUG:teuthology.parallel:result is None 2024-02-23T00:14:09.657 DEBUG:teuthology.parallel:result is None 2024-02-23T00:14:09.657 INFO:teuthology.run_tasks:Running task pexec... 2024-02-23T00:14:09.665 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-23T00:14:09.666 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-02-23T00:14:09.666 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:14:09.667 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi118.front.sepia.ceph.com 2024-02-23T00:14:09.667 DEBUG:teuthology.orchestra.run.smithi118:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:14:09.857 DEBUG:teuthology.parallel:result is None 2024-02-23T00:14:09.893 DEBUG:teuthology.parallel:result is None 2024-02-23T00:14:09.893 INFO:teuthology.run_tasks:Running task install... 2024-02-23T00:14:09.902 DEBUG:teuthology.task.install:project ceph 2024-02-23T00:14:09.903 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'}} 2024-02-23T00:14:09.903 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'} 2024-02-23T00:14:09.903 INFO:teuthology.task.install:Using flavor: default 2024-02-23T00:14:09.911 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-23T00:14:09.911 INFO:teuthology.task.install:extra packages: [] 2024-02-23T00:14:09.911 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-23T00:14:09.911 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:14:09.911 INFO:teuthology.packaging:ref: None 2024-02-23T00:14:09.911 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:14:09.912 INFO:teuthology.packaging:branch: None 2024-02-23T00:14:09.912 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:14:10.106 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-23T00:14:10.106 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-23T00:14:10.109 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-23T00:14:10.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:14:10.110 INFO:teuthology.packaging:ref: None 2024-02-23T00:14:10.110 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:14:10.110 INFO:teuthology.packaging:branch: None 2024-02-23T00:14:10.110 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:14:10.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-23T00:14:10.282 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-23T00:14:10.282 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-23T00:14:10.323 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-23T00:14:10.324 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:14:10.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-23T00:14:10.351 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-23T00:14:10.351 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:14:10.352 INFO:teuthology.packaging:ref: None 2024-02-23T00:14:10.352 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:14:10.352 INFO:teuthology.packaging:branch: None 2024-02-23T00:14:10.352 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:14:10.352 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-23T00:14:10.362 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-23T00:14:10.362 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-23T00:14:10.395 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-23T00:14:10.395 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:14:10.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-23T00:14:10.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-23T00:14:10.423 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-23T00:14:10.424 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:14:10.424 INFO:teuthology.packaging:ref: None 2024-02-23T00:14:10.424 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:14:10.424 INFO:teuthology.packaging:branch: None 2024-02-23T00:14:10.424 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:14:10.424 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-23T00:14:10.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-23T00:14:10.495 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-23T00:14:10.561 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-02-23T00:14:10.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-02-23T00:14:10.568 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-23T00:14:10.635 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-02-23T00:14:10.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-02-23T00:14:10.824 INFO:teuthology.orchestra.run.smithi118.stdout:76 files removed 2024-02-23T00:14:10.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-02-23T00:14:10.900 INFO:teuthology.orchestra.run.smithi090.stdout:76 files removed 2024-02-23T00:14:10.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-02-23T00:14:11.330 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 977 kB/s | 74 kB 00:00 2024-02-23T00:14:11.448 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-02-23T00:14:11.460 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 580 kB/s | 74 kB 00:00 2024-02-23T00:14:11.577 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-02-23T00:14:11.583 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 16 kB/s | 1.7 kB 00:00 2024-02-23T00:14:11.686 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-02-23T00:14:15.921 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - AppStream 6.5 MB/s | 27 MB 00:04 2024-02-23T00:14:16.816 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - AppStream 5.3 MB/s | 27 MB 00:05 2024-02-23T00:14:21.657 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-23T00:14:22.458 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-23T00:14:23.616 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2024-02-23T00:14:24.083 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-23T00:14:24.426 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-02-23T00:14:24.780 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - Extras common packages 29 kB/s | 6.9 kB 00:00 2024-02-23T00:18:44.674 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 8 - PowerTools 19 kB/s | 4.9 MB 04:20 2024-02-23T00:18:45.717 INFO:teuthology.orchestra.run.smithi090.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-23T00:18:47.485 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 8 - PowerTools 19 kB/s | 4.9 MB 04:22 2024-02-23T00:18:48.552 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-23T00:18:49.889 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-02-23T00:18:53.712 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 185 kB/s | 24 kB 00:00 2024-02-23T00:18:57.292 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-23T00:18:57.295 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-23T00:18:57.296 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-23T00:18:57.297 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout:Install 25 Packages 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-02-23T00:18:57.298 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:18:57.299 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 59 M 2024-02-23T00:18:57.299 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:18:58.030 INFO:teuthology.orchestra.run.smithi118.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2024-02-23T00:18:58.055 INFO:teuthology.orchestra.run.smithi118.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-02-23T00:18:58.097 INFO:teuthology.orchestra.run.smithi118.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-23T00:18:58.130 INFO:teuthology.orchestra.run.smithi118.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-23T00:18:58.231 INFO:teuthology.orchestra.run.smithi118.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-23T00:18:58.256 INFO:teuthology.orchestra.run.smithi118.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-02-23T00:18:58.281 INFO:teuthology.orchestra.run.smithi118.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-23T00:18:58.315 INFO:teuthology.orchestra.run.smithi118.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-02-23T00:18:58.382 INFO:teuthology.orchestra.run.smithi118.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 5.7 MB/s | 387 kB 00:00 2024-02-23T00:18:58.416 INFO:teuthology.orchestra.run.smithi118.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-23T00:18:58.441 INFO:teuthology.orchestra.run.smithi118.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-23T00:18:58.617 INFO:teuthology.orchestra.run.smithi118.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-02-23T00:18:58.642 INFO:teuthology.orchestra.run.smithi118.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 725 kB/s | 145 kB 00:00 2024-02-23T00:18:58.742 INFO:teuthology.orchestra.run.smithi118.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-02-23T00:18:59.009 INFO:teuthology.orchestra.run.smithi118.stdout:(15/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2024-02-23T00:18:59.034 INFO:teuthology.orchestra.run.smithi118.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 150 kB/s | 44 kB 00:00 2024-02-23T00:18:59.110 INFO:teuthology.orchestra.run.smithi118.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-02-23T00:18:59.210 INFO:teuthology.orchestra.run.smithi118.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 211 kB/s | 42 kB 00:00 2024-02-23T00:18:59.244 INFO:teuthology.orchestra.run.smithi118.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 225 kB/s | 47 kB 00:00 2024-02-23T00:18:59.269 INFO:teuthology.orchestra.run.smithi118.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 505 kB/s | 80 kB 00:00 2024-02-23T00:18:59.620 INFO:teuthology.orchestra.run.smithi118.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 517 kB/s | 181 kB 00:00 2024-02-23T00:18:59.761 INFO:teuthology.orchestra.run.smithi118.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 415 kB/s | 59 kB 00:00 2024-02-23T00:18:59.820 INFO:teuthology.orchestra.run.smithi118.stdout:(23/27): fmt-6.2.1-1.el8.x86_64.rpm 191 kB/s | 116 kB 00:00 2024-02-23T00:18:59.887 INFO:teuthology.orchestra.run.smithi118.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 56 MB/s | 3.7 MB 00:00 2024-02-23T00:18:59.912 INFO:teuthology.orchestra.run.smithi118.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 498 kB/s | 75 kB 00:00 2024-02-23T00:18:59.946 INFO:teuthology.orchestra.run.smithi118.stdout:(26/27): gperftools-libs-2.7-9.el8.x86_64.rpm 436 kB/s | 306 kB 00:00 2024-02-23T00:19:00.013 INFO:teuthology.orchestra.run.smithi118.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.0 MB 00:00 2024-02-23T00:19:00.015 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:00.015 INFO:teuthology.orchestra.run.smithi118.stdout:Total 22 MB/s | 59 MB 00:02 2024-02-23T00:19:00.081 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:19:00.130 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:19:00.130 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:19:00.481 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:19:00.484 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:19:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:19:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-23T00:19:02.138 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-23T00:19:02.178 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:16 2024-02-23T00:19:02.346 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-23T00:19:02.769 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-23T00:19:03.013 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:03.384 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-23T00:19:03.597 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:03.660 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-23T00:19:04.470 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-23T00:19:04.671 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:04.968 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-23T00:19:05.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-23T00:19:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-23T00:19:05.410 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-23T00:19:05.622 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-23T00:19:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-23T00:19:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-23T00:19:05.909 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 346 kB/s | 24 kB 00:00 2024-02-23T00:19:06.095 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:06.516 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:06.771 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-23T00:19:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-23T00:19:07.083 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-23T00:19:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-23T00:19:07.470 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-23T00:19:07.543 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-23T00:19:07.747 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-23T00:19:07.979 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-23T00:19:08.206 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-23T00:19:08.404 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-23T00:19:09.482 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-23T00:19:09.485 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-23T00:19:09.486 INFO:teuthology.orchestra.run.smithi090.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout:Install 25 Packages 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-02-23T00:19:09.487 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:09.488 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 59 M 2024-02-23T00:19:09.488 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:19:11.048 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:11.329 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:11.670 INFO:teuthology.orchestra.run.smithi090.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2024-02-23T00:19:11.695 INFO:teuthology.orchestra.run.smithi090.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-02-23T00:19:11.770 INFO:teuthology.orchestra.run.smithi090.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 10 MB/s | 801 kB 00:00 2024-02-23T00:19:11.814 INFO:teuthology.orchestra.run.smithi090.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2024-02-23T00:19:11.922 INFO:teuthology.orchestra.run.smithi090.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2024-02-23T00:19:11.947 INFO:teuthology.orchestra.run.smithi090.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-23T00:19:11.972 INFO:teuthology.orchestra.run.smithi090.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-23T00:19:12.006 INFO:teuthology.orchestra.run.smithi090.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-02-23T00:19:12.037 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:12.040 INFO:teuthology.orchestra.run.smithi090.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-23T00:19:12.065 INFO:teuthology.orchestra.run.smithi090.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-02-23T00:19:12.099 INFO:teuthology.orchestra.run.smithi090.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 114 kB 00:00 2024-02-23T00:19:12.275 INFO:teuthology.orchestra.run.smithi090.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-02-23T00:19:12.300 INFO:teuthology.orchestra.run.smithi090.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 725 kB/s | 145 kB 00:00 2024-02-23T00:19:12.369 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-23T00:19:12.509 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-23T00:19:12.600 INFO:teuthology.orchestra.run.smithi090.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2024-02-23T00:19:12.634 INFO:teuthology.orchestra.run.smithi090.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 778 kB/s | 259 kB 00:00 2024-02-23T00:19:12.645 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:12.709 INFO:teuthology.orchestra.run.smithi090.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.5 MB/s | 677 kB 00:00 2024-02-23T00:19:12.734 INFO:teuthology.orchestra.run.smithi090.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 328 kB/s | 44 kB 00:00 2024-02-23T00:19:12.760 INFO:teuthology.orchestra.run.smithi090.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 336 kB/s | 42 kB 00:00 2024-02-23T00:19:12.860 INFO:teuthology.orchestra.run.smithi090.stdout:(19/27): fmt-6.2.1-1.el8.x86_64.rpm 1.1 MB/s | 116 kB 00:00 2024-02-23T00:19:12.885 INFO:teuthology.orchestra.run.smithi090.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 532 kB/s | 80 kB 00:00 2024-02-23T00:19:12.911 INFO:teuthology.orchestra.run.smithi090.stdout:(21/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2024-02-23T00:19:12.944 INFO:teuthology.orchestra.run.smithi090.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 3.6 MB/s | 306 kB 00:00 2024-02-23T00:19:12.986 INFO:teuthology.orchestra.run.smithi090.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2024-02-23T00:19:13.028 INFO:teuthology.orchestra.run.smithi090.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2024-02-23T00:19:13.053 INFO:teuthology.orchestra.run.smithi090.stdout:(25/27): liboath-2.6.2-3.el8.x86_64.rpm 426 kB/s | 59 kB 00:00 2024-02-23T00:19:13.137 INFO:teuthology.orchestra.run.smithi090.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-02-23T00:19:13.221 INFO:teuthology.orchestra.run.smithi090.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-23T00:19:13.225 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:13.226 INFO:teuthology.orchestra.run.smithi090.stdout:Total 16 MB/s | 59 MB 00:03 2024-02-23T00:19:13.295 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:19:13.343 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:19:13.344 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:19:13.681 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:19:13.684 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:19:14.799 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:19:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-23T00:19:15.289 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-23T00:19:15.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-23T00:19:15.851 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-23T00:19:16.046 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:16.165 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:16.411 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-23T00:19:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-23T00:19:17.411 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-23T00:19:17.588 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-23T00:19:18.013 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-23T00:19:18.085 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-23T00:19:18.251 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-23T00:19:18.480 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-23T00:19:18.771 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-23T00:19:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-23T00:19:19.023 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:19.663 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-23T00:19:19.762 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-23T00:19:19.949 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-23T00:19:20.173 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-23T00:19:20.329 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-23T00:19:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-23T00:19:20.649 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-23T00:19:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-23T00:19:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-23T00:19:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-23T00:19:23.847 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-23T00:19:25.073 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-23T00:19:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-23T00:19:25.321 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /run 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-02-23T00:19:32.013 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-02-23T00:19:32.014 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2024-02-23T00:19:32.014 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:33.407 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-23T00:19:33.486 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-23T00:19:33.487 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-23T00:19:33.712 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-23T00:19:33.712 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:34.360 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-23T00:19:34.361 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-23T00:19:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.356 INFO:teuthology.orchestra.run.smithi118.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb-1.22-1.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-23T00:19:35.357 INFO:teuthology.orchestra.run.smithi118.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:35.358 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:19:35.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-02-23T00:19:36.410 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:43 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:19:37.935 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-23T00:19:37.936 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-02-23T00:19:37.937 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:37.938 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 53 M 2024-02-23T00:19:37.938 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 229 M 2024-02-23T00:19:37.938 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:19:38.296 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 169 kB 00:00 2024-02-23T00:19:38.321 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-02-23T00:19:38.388 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-02-23T00:19:38.455 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-02-23T00:19:38.480 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-02-23T00:19:39.047 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 57 MB/s | 52 MB 00:00 2024-02-23T00:19:39.049 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:39.049 INFO:teuthology.orchestra.run.smithi118.stdout:Total 48 MB/s | 53 MB 00:01 2024-02-23T00:19:39.091 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:19:39.112 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:19:39.112 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:19:39.443 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:19:39.446 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:19:39.970 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:19:40.185 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-23T00:19:40.416 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-23T00:19:40.574 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:19:40.651 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:19:40.856 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-23T00:19:41.030 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-23T00:19:45.024 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:45.024 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-02-23T00:19:45.024 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /run 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /dev 2024-02-23T00:19:45.025 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:45.885 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-23T00:19:46.155 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-23T00:19:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-23T00:19:46.425 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-23T00:19:46.426 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-23T00:19:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-23T00:19:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-23T00:19:46.697 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:46.942 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:19:47.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-23T00:19:47.400 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-23T00:19:47.401 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-23T00:19:47.402 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-23T00:19:47.907 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:54 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:19:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-23T00:19:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:48.391 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-02-23T00:19:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.391 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: leveldb-1.22-1.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:48.393 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:19:48.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-02-23T00:19:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:44 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:19:49.450 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:19:49.455 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:49.455 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:19:49.455 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:49.455 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-23T00:19:49.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-23T00:19:49.457 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:49.458 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout:Install 32 Packages 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 M 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 111 M 2024-02-23T00:19:49.459 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:19:50.102 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 64 kB/s | 6.3 kB 00:00 2024-02-23T00:19:50.159 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-02-23T00:19:50.217 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-02-23T00:19:50.251 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-23T00:19:50.335 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-23T00:19:50.610 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2024-02-23T00:19:50.635 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 129 kB/s | 39 kB 00:00 2024-02-23T00:19:50.660 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 384 kB/s | 157 kB 00:00 2024-02-23T00:19:50.703 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 849 kB/s | 57 kB 00:00 2024-02-23T00:19:50.744 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 773 kB/s | 103 kB 00:00 2024-02-23T00:19:50.794 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2024-02-23T00:19:50.987 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 569 kB/s | 109 kB 00:00 2024-02-23T00:19:50.990 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-23T00:19:50.992 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 53 M 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 229 M 2024-02-23T00:19:50.993 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:19:51.012 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 805 kB/s | 249 kB 00:00 2024-02-23T00:19:51.105 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-02-23T00:19:51.180 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.4 MB/s | 203 kB 00:00 2024-02-23T00:19:51.237 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.3 MB/s | 641 kB 00:00 2024-02-23T00:19:51.239 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-02-23T00:19:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-02-23T00:19:51.289 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-23T00:19:51.463 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 92 kB/s | 44 kB 00:00 2024-02-23T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 389 kB/s | 185 kB 00:00 2024-02-23T00:19:51.613 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-02-23T00:19:51.647 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 422 kB/s | 173 kB 00:00 2024-02-23T00:19:51.723 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-23T00:19:51.748 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 486 kB/s | 49 kB 00:00 2024-02-23T00:19:51.823 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-02-23T00:19:51.848 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 467 kB/s | 59 kB 00:00 2024-02-23T00:19:51.924 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 58 MB/s | 52 MB 00:00 2024-02-23T00:19:51.925 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:51.925 INFO:teuthology.orchestra.run.smithi090.stdout:Total 57 MB/s | 53 MB 00:00 2024-02-23T00:19:51.940 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-02-23T00:19:51.957 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 116 kB/s | 16 kB 00:00 2024-02-23T00:19:51.967 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:19:51.982 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 139 kB/s | 19 kB 00:00 2024-02-23T00:19:51.987 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:19:51.988 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:19:52.091 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 254 kB/s | 27 kB 00:00 2024-02-23T00:19:52.141 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): python3-singledispatch-3.4.0.3-18.el8. 119 kB/s | 24 kB 00:00 2024-02-23T00:19:52.167 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): python3-tempora-1.14.1-5.el8.noarch.rp 140 kB/s | 29 kB 00:00 2024-02-23T00:19:52.299 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:19:52.300 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2024-02-23T00:19:52.302 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:19:52.325 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 539 kB/s | 85 kB 00:00 2024-02-23T00:19:52.351 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-02-23T00:19:52.409 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-02-23T00:19:52.410 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:52.411 INFO:teuthology.orchestra.run.smithi118.stdout:Total 10 MB/s | 31 MB 00:02 2024-02-23T00:19:52.489 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:19:52.527 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:19:52.527 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:19:52.786 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:19:52.843 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:19:52.846 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:19:52.973 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-23T00:19:53.139 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-23T00:19:53.336 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:19:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:19:53.617 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-23T00:19:53.786 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-23T00:19:54.016 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:19:54.252 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-23T00:19:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-23T00:19:54.715 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-23T00:19:54.945 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-23T00:19:55.118 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-23T00:19:55.332 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-23T00:19:55.492 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-23T00:19:55.702 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-23T00:19:55.894 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-23T00:19:56.132 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-23T00:19:56.350 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-23T00:19:56.593 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-23T00:19:56.995 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-23T00:19:57.332 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-23T00:19:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-23T00:19:57.772 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-23T00:19:57.980 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:19:58.297 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-23T00:19:58.510 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-23T00:19:58.667 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-23T00:19:58.723 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-23T00:19:58.893 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-23T00:19:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-23T00:19:59.176 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-23T00:19:59.176 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-23T00:19:59.176 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-23T00:19:59.176 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-23T00:19:59.176 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-23T00:19:59.177 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-23T00:19:59.212 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-23T00:19:59.474 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-23T00:19:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-23T00:19:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:59.636 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:19:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-23T00:19:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-23T00:19:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-23T00:19:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:19:59.637 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:19:59.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-23T00:19:59.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-02-23T00:20:00.085 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-23T00:20:00.419 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-23T00:20:00.647 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:55 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:00.799 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-23T00:20:00.994 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-23T00:20:02.226 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:20:02.229 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-23T00:20:02.230 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-23T00:20:02.231 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-23T00:20:02.232 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:02.233 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:02.234 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:02.234 INFO:teuthology.orchestra.run.smithi090.stdout:Install 32 Packages 2024-02-23T00:20:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:02.235 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 M 2024-02-23T00:20:02.235 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 111 M 2024-02-23T00:20:02.235 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:02.555 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-23T00:20:02.827 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-23T00:20:03.260 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-23T00:20:03.527 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-23T00:20:03.748 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-23T00:20:03.946 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-23T00:20:03.963 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-23T00:20:04.128 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-23T00:20:04.541 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-23T00:20:04.542 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-23T00:20:04.543 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-23T00:20:04.640 INFO:teuthology.orchestra.run.smithi090.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 127 kB/s | 6.3 kB 00:00 2024-02-23T00:20:04.699 INFO:teuthology.orchestra.run.smithi090.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-02-23T00:20:04.766 INFO:teuthology.orchestra.run.smithi090.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-23T00:20:04.791 INFO:teuthology.orchestra.run.smithi090.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-23T00:20:04.892 INFO:teuthology.orchestra.run.smithi090.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-02-23T00:20:05.203 INFO:teuthology.orchestra.run.smithi090.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 36 MB/s | 18 MB 00:00 2024-02-23T00:20:05.234 INFO:teuthology.orchestra.run.smithi090.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 113 kB/s | 39 kB 00:00 2024-02-23T00:20:05.259 INFO:teuthology.orchestra.run.smithi090.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 336 kB/s | 157 kB 00:00 2024-02-23T00:20:05.293 INFO:teuthology.orchestra.run.smithi090.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2024-02-23T00:20:05.360 INFO:teuthology.orchestra.run.smithi090.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 454 kB/s | 57 kB 00:00 2024-02-23T00:20:05.403 INFO:teuthology.orchestra.run.smithi090.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.1 MB/s | 457 kB 00:00 2024-02-23T00:20:05.442 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-23T00:20:05.442 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:05.442 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:05.442 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-23T00:20:05.443 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-2.6-1.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-23T00:20:05.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-23T00:20:05.445 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-23T00:20:05.445 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-23T00:20:05.445 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:05.445 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:05.604 INFO:teuthology.orchestra.run.smithi090.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 802 kB/s | 249 kB 00:00 2024-02-23T00:20:05.629 INFO:teuthology.orchestra.run.smithi090.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2024-02-23T00:20:05.662 INFO:teuthology.orchestra.run.smithi090.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.1 MB/s | 641 kB 00:00 2024-02-23T00:20:05.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-02-23T00:20:05.905 INFO:teuthology.orchestra.run.smithi090.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 145 kB/s | 44 kB 00:00 2024-02-23T00:20:05.988 INFO:teuthology.orchestra.run.smithi090.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 516 kB/s | 185 kB 00:00 2024-02-23T00:20:06.013 INFO:teuthology.orchestra.run.smithi090.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 492 kB/s | 173 kB 00:00 2024-02-23T00:20:06.064 INFO:teuthology.orchestra.run.smithi090.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2024-02-23T00:20:06.089 INFO:teuthology.orchestra.run.smithi090.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-02-23T00:20:06.148 INFO:teuthology.orchestra.run.smithi090.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 582 kB/s | 49 kB 00:00 2024-02-23T00:20:06.181 INFO:teuthology.orchestra.run.smithi090.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-02-23T00:20:06.207 INFO:teuthology.orchestra.run.smithi090.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 500 kB/s | 59 kB 00:00 2024-02-23T00:20:06.282 INFO:teuthology.orchestra.run.smithi090.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2024-02-23T00:20:06.307 INFO:teuthology.orchestra.run.smithi090.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-23T00:20:06.332 INFO:teuthology.orchestra.run.smithi090.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2024-02-23T00:20:06.358 INFO:teuthology.orchestra.run.smithi090.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2024-02-23T00:20:06.392 INFO:teuthology.orchestra.run.smithi090.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-02-23T00:20:06.417 INFO:teuthology.orchestra.run.smithi090.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 327 kB/s | 27 kB 00:00 2024-02-23T00:20:06.476 INFO:teuthology.orchestra.run.smithi090.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.1 MB/s | 255 kB 00:00 2024-02-23T00:20:06.500 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:13 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:06.509 INFO:teuthology.orchestra.run.smithi090.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2024-02-23T00:20:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 727 kB/s | 85 kB 00:00 2024-02-23T00:20:06.560 INFO:teuthology.orchestra.run.smithi090.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2024-02-23T00:20:06.564 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:06.564 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.2 MB/s | 31 MB 00:04 2024-02-23T00:20:06.645 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:06.993 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:06.996 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:08.022 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:20:08.069 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:08.070 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:20:08.070 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:08.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-02-23T00:20:08.308 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-23T00:20:08.538 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-23T00:20:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-23T00:20:08.912 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:15 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:08.967 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-23T00:20:09.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-23T00:20:09.303 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-23T00:20:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-23T00:20:09.641 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-23T00:20:09.831 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-23T00:20:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-23T00:20:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-23T00:20:10.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-23T00:20:10.473 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:10.473 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:10.473 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:10.473 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:10.473 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 71 k 2024-02-23T00:20:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 290 k 2024-02-23T00:20:10.475 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:10.534 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.2 MB/s | 71 kB 00:00 2024-02-23T00:20:10.534 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:10.535 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.1 MB/s | 71 kB 00:00 2024-02-23T00:20:10.535 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:10.542 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:10.542 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:10.543 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:10.547 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-23T00:20:11.088 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-23T00:20:11.097 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-23T00:20:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-23T00:20:11.483 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:11.645 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:20:11.885 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:11.886 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:11.886 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:11.886 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-23T00:20:11.886 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:11.886 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-23T00:20:12.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-02-23T00:20:12.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-23T00:20:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-23T00:20:12.499 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-23T00:20:12.741 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-23T00:20:12.844 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:19 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-23T00:20:13.162 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-23T00:20:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-23T00:20:13.759 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-23T00:20:14.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-23T00:20:14.346 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-23T00:20:14.430 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:14.430 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:20:14.431 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:14.432 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 163 k 2024-02-23T00:20:14.432 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 587 k 2024-02-23T00:20:14.432 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:14.474 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.8 MB/s | 163 kB 00:00 2024-02-23T00:20:14.474 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:14.475 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.7 MB/s | 163 kB 00:00 2024-02-23T00:20:14.475 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:14.485 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:14.485 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:14.532 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:14.535 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:14.671 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:15.331 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:15.735 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-23T00:20:15.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-02-23T00:20:16.162 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-23T00:20:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-23T00:20:16.683 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:23 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:16.829 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-23T00:20:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-23T00:20:17.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-23T00:20:17.279 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-23T00:20:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-23T00:20:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-23T00:20:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-23T00:20:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-23T00:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-23T00:20:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-23T00:20:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-23T00:20:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-23T00:20:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-23T00:20:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-23T00:20:18.213 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:20:18.260 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:18.261 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:20:18.261 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:18.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:18.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-23T00:20:18.706 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-2.6-1.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-23T00:20:18.707 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:18.708 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:18.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-02-23T00:20:19.144 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:26 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:19.736 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:14 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:20.714 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:20.715 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:20.715 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:20:20.715 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:20.716 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-02-23T00:20:20.717 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:20.717 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.0 M 2024-02-23T00:20:20.717 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 15 M 2024-02-23T00:20:20.717 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:21.012 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 534 kB/s | 24 kB 00:00 2024-02-23T00:20:21.037 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 127 kB/s | 8.9 kB 00:00 2024-02-23T00:20:21.104 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-02-23T00:20:21.246 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2024-02-23T00:20:21.272 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:20:21.318 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:21.319 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:20:21.320 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:21.397 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 94 kB/s | 34 kB 00:00 2024-02-23T00:20:21.469 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-02-23T00:20:21.655 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 357 kB/s | 196 kB 00:00 2024-02-23T00:20:21.657 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:21.657 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.1 MB/s | 3.0 MB 00:00 2024-02-23T00:20:21.667 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:21.677 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:21.677 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:21.844 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:21.845 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:22.140 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:22.215 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:17 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:22.382 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-23T00:20:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-23T00:20:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-23T00:20:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-23T00:20:23.780 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:23.781 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 71 k 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 290 k 2024-02-23T00:20:23.782 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:23.837 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-02-23T00:20:23.838 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:23.838 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.2 MB/s | 71 kB 00:00 2024-02-23T00:20:23.838 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:23.845 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:23.845 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:23.847 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:23.850 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:23.889 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-23T00:20:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-23T00:20:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-23T00:20:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-23T00:20:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-23T00:20:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-23T00:20:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-23T00:20:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-23T00:20:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-23T00:20:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-23T00:20:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-23T00:20:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:25.034 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:25.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:25.180 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:25.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-02-23T00:20:25.895 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:32 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:26.137 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:21 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:27.469 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-02-23T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-02-23T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:27.470 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:27.471 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-02-23T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout:Install 8 Packages 2024-02-23T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:27.472 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2024-02-23T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 210 M 2024-02-23T00:20:27.473 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:27.701 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:27.701 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:27.701 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:20:27.702 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:27.703 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 163 k 2024-02-23T00:20:27.703 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 587 k 2024-02-23T00:20:27.703 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:27.860 INFO:teuthology.orchestra.run.smithi118.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-02-23T00:20:27.904 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 810 kB/s | 163 kB 00:00 2024-02-23T00:20:27.904 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:27.905 INFO:teuthology.orchestra.run.smithi090.stdout:Total 804 kB/s | 163 kB 00:00 2024-02-23T00:20:27.905 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:27.917 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:27.917 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:27.965 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:27.971 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:28.106 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:28.594 INFO:teuthology.orchestra.run.smithi118.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.4 MB/s | 4.7 MB 00:00 2024-02-23T00:20:28.661 INFO:teuthology.orchestra.run.smithi118.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.9 MB 00:00 2024-02-23T00:20:28.686 INFO:teuthology.orchestra.run.smithi118.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-23T00:20:28.711 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:28.820 INFO:teuthology.orchestra.run.smithi118.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.9 MB/s | 3.7 MB 00:00 2024-02-23T00:20:29.078 INFO:teuthology.orchestra.run.smithi118.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 645 kB 00:00 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:29.079 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:29.137 INFO:teuthology.orchestra.run.smithi118.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 544 kB/s | 172 kB 00:00 2024-02-23T00:20:29.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-02-23T00:20:29.604 INFO:teuthology.orchestra.run.smithi118.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-02-23T00:20:29.605 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:29.605 INFO:teuthology.orchestra.run.smithi118.stdout:Total 17 MB/s | 37 MB 00:02 2024-02-23T00:20:29.835 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:29.853 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:29.853 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:30.067 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:25 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:30.181 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:30.184 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:30.821 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:30.949 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:20:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:20:31.282 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:20:31.588 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:20:31.635 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:31.636 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:20:31.636 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:31.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-02-23T00:20:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:20:32.091 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:32.568 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:27 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:32.738 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:32.965 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:20:33.664 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:20:34.027 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:20:34.138 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-23T00:20:34.139 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-23T00:20:34.140 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:34.141 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.0 M 2024-02-23T00:20:34.141 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 15 M 2024-02-23T00:20:34.141 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:34.257 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:20:34.287 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 492 kB/s | 24 kB 00:00 2024-02-23T00:20:34.313 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 120 kB/s | 8.9 kB 00:00 2024-02-23T00:20:34.373 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-02-23T00:20:34.473 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 230 kB/s | 43 kB 00:00 2024-02-23T00:20:34.748 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 77 kB/s | 34 kB 00:00 2024-02-23T00:20:35.073 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 280 kB/s | 196 kB 00:00 2024-02-23T00:20:35.076 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:35.076 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.2 MB/s | 3.0 MB 00:00 2024-02-23T00:20:35.087 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:35.097 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:35.097 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:35.265 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:35.268 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:35.732 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-23T00:20:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-23T00:20:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:20:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-23T00:20:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-23T00:20:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-23T00:20:37.471 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-23T00:20:37.587 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:37.703 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-23T00:20:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-23T00:20:38.022 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-23T00:20:38.023 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-23T00:20:38.236 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-23T00:20:38.236 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:38.237 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:38.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-23T00:20:38.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:38.485 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:38.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-02-23T00:20:39.124 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:34 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:39.355 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:40.671 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-23T00:20:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout:Install 8 Packages 2024-02-23T00:20:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:40.674 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2024-02-23T00:20:40.674 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 210 M 2024-02-23T00:20:40.674 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:40.921 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:40.923 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:40.923 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:20:40.923 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:40.923 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-23T00:20:40.924 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-02-23T00:20:40.925 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:40.926 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 1.9 M 2024-02-23T00:20:40.926 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 24 M 2024-02-23T00:20:40.926 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:41.248 INFO:teuthology.orchestra.run.smithi090.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 43 MB/s | 8.6 MB 00:00 2024-02-23T00:20:41.330 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 502 kB/s | 27 kB 00:00 2024-02-23T00:20:41.397 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 166 kB/s | 20 kB 00:00 2024-02-23T00:20:41.422 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 182 kB/s | 27 kB 00:00 2024-02-23T00:20:41.465 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 943 kB/s | 126 kB 00:00 2024-02-23T00:20:41.489 INFO:teuthology.orchestra.run.smithi090.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-02-23T00:20:41.490 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-23T00:20:41.567 INFO:teuthology.orchestra.run.smithi090.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 9.1 MB/s | 4.7 MB 00:00 2024-02-23T00:20:41.600 INFO:teuthology.orchestra.run.smithi090.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-02-23T00:20:41.650 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 682 kB/s | 155 kB 00:00 2024-02-23T00:20:41.674 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 204 kB/s | 43 kB 00:00 2024-02-23T00:20:41.690 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 164 kB/s | 33 kB 00:00 2024-02-23T00:20:41.841 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 85 kB/s | 16 kB 00:00 2024-02-23T00:20:41.908 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 348 kB/s | 81 kB 00:00 2024-02-23T00:20:41.950 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 592 kB/s | 64 kB 00:00 2024-02-23T00:20:41.966 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-02-23T00:20:41.984 INFO:teuthology.orchestra.run.smithi090.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 645 kB 00:00 2024-02-23T00:20:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 22 MB/s | 14 MB 00:00 2024-02-23T00:20:42.258 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-02-23T00:20:42.261 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:42.261 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-02-23T00:20:42.284 INFO:teuthology.orchestra.run.smithi090.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.6 MB/s | 3.7 MB 00:01 2024-02-23T00:20:42.320 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:42.326 INFO:teuthology.orchestra.run.smithi090.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 504 kB/s | 172 kB 00:00 2024-02-23T00:20:42.327 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:42.327 INFO:teuthology.orchestra.run.smithi090.stdout:Total 22 MB/s | 37 MB 00:01 2024-02-23T00:20:42.338 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:42.338 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:42.432 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:42.435 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:42.557 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:42.575 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:42.575 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:42.900 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:42.903 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:42.987 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:43.242 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-23T00:20:43.431 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-23T00:20:43.507 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:20:43.709 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-23T00:20:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:20:43.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:20:43.988 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-23T00:20:44.195 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-23T00:20:44.454 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-23T00:20:44.741 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:20:44.771 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-23T00:20:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:45.036 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-23T00:20:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:20:46.076 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-23T00:20:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-23T00:20:46.464 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:20:46.620 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-23T00:20:46.796 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:20:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-23T00:20:47.015 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-23T00:20:47.050 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:20:47.304 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-23T00:20:47.304 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-23T00:20:47.304 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-23T00:20:47.304 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-23T00:20:47.304 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-23T00:20:47.305 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-23T00:20:47.778 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-23T00:20:47.778 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:47.778 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:47.778 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-23T00:20:47.778 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:47.779 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:47.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-02-23T00:20:48.671 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:55 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:20:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-23T00:20:50.233 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-23T00:20:50.234 INFO:teuthology.orchestra.run.smithi118.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-02-23T00:20:50.235 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:50.236 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 5.9 M 2024-02-23T00:20:50.236 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 24 M 2024-02-23T00:20:50.236 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:50.784 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-02-23T00:20:50.918 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2024-02-23T00:20:50.985 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.3 MB/s | 578 kB 00:00 2024-02-23T00:20:51.076 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.2 MB/s | 355 kB 00:00 2024-02-23T00:20:51.149 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-23T00:20:51.169 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2024-02-23T00:20:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-23T00:20:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-23T00:20:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-23T00:20:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:20:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-23T00:20:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:20:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-23T00:20:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-23T00:20:51.752 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.7 MB/s | 4.8 MB 00:01 2024-02-23T00:20:51.754 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:51.754 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.9 MB/s | 5.9 MB 00:01 2024-02-23T00:20:51.806 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:51.816 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:51.816 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-23T00:20:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-23T00:20:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:51.878 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:20:51.921 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:51.925 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:51.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-02-23T00:20:52.192 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:52.462 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-23T00:20:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:20:52.807 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-23T00:20:53.811 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-23T00:20:54.199 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-23T00:20:54.328 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-23T00:20:54.330 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:20:54.331 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-02-23T00:20:54.332 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:20:54.332 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.9 M 2024-02-23T00:20:54.332 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2024-02-23T00:20:54.332 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:20:54.475 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-23T00:20:54.626 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 345 kB/s | 27 kB 00:00 2024-02-23T00:20:54.678 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:54.701 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 174 kB/s | 27 kB 00:00 2024-02-23T00:20:54.726 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 112 kB/s | 20 kB 00:00 2024-02-23T00:20:54.769 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 888 kB/s | 126 kB 00:00 2024-02-23T00:20:54.794 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-23T00:20:54.894 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 340 kB/s | 43 kB 00:00 2024-02-23T00:20:54.921 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 799 kB/s | 155 kB 00:00 2024-02-23T00:20:54.981 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:20:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-23T00:20:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-23T00:20:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-23T00:20:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-23T00:20:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-23T00:20:55.021 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 145 kB/s | 33 kB 00:00 2024-02-23T00:20:55.105 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 78 kB/s | 16 kB 00:00 2024-02-23T00:20:55.205 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 286 kB/s | 81 kB 00:00 2024-02-23T00:20:55.247 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 452 kB/s | 64 kB 00:00 2024-02-23T00:20:55.297 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 666 kB/s | 61 kB 00:00 2024-02-23T00:20:55.348 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.6 MB/s | 1.2 MB 00:00 2024-02-23T00:20:55.350 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:55.350 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2024-02-23T00:20:55.407 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:20:55.423 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:20:55.423 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-23T00:20:55.454 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:55.455 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:20:55.510 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:20:55.513 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:20:55.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-02-23T00:20:56.055 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:20:56.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-23T00:20:56.336 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:03 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:20:56.549 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-23T00:20:56.744 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-23T00:20:57.024 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-23T00:20:57.231 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-23T00:20:57.454 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-23T00:20:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-23T00:20:57.901 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:20:57.902 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:57.902 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:57.902 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:57.902 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-02-23T00:20:57.903 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:57.904 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 968 k 2024-02-23T00:20:57.904 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 3.2 M 2024-02-23T00:20:57.904 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:20:57.971 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-23T00:20:58.147 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-23T00:20:58.289 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 416 kB/s | 83 kB 00:00 2024-02-23T00:20:58.290 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:58.290 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.4 MB/s | 968 kB 00:00 2024-02-23T00:20:58.299 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:20:58.320 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:20:58.320 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:20:58.405 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:20:58.408 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:20:58.599 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:20:58.886 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-23T00:20:58.982 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-23T00:20:59.111 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-23T00:20:59.265 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-23T00:20:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-23T00:20:59.502 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-23T00:20:59.502 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-23T00:20:59.674 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-23T00:20:59.866 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-23T00:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-23T00:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-23T00:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:00.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-02-23T00:21:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-23T00:21:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-23T00:21:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:00.623 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:00.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-02-23T00:21:00.914 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:07 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:01.507 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:56 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:02.481 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:21:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:02.483 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 131 k 2024-02-23T00:21:02.483 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 439 k 2024-02-23T00:21:02.483 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:21:02.536 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-02-23T00:21:02.537 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:02.537 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-02-23T00:21:02.537 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:21:02.542 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:21:02.542 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:21:02.590 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:21:02.593 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:21:02.698 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:21:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:03.066 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-23T00:21:03.067 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-02-23T00:21:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:03.069 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.9 M 2024-02-23T00:21:03.069 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 24 M 2024-02-23T00:21:03.069 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:03.601 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:03.677 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:03.678 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:21:03.678 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:03.678 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:03.678 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:03.678 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-02-23T00:21:03.787 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.8 MB/s | 578 kB 00:00 2024-02-23T00:21:03.792 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-02-23T00:21:04.003 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 56 kB/s | 21 kB 00:00 2024-02-23T00:21:04.103 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.2 MB/s | 4.8 MB 00:00 2024-02-23T00:21:04.204 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 244 kB/s | 49 kB 00:00 2024-02-23T00:21:04.470 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 519 kB/s | 355 kB 00:00 2024-02-23T00:21:04.472 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:04.472 INFO:teuthology.orchestra.run.smithi090.stdout:Total 4.2 MB/s | 5.9 MB 00:01 2024-02-23T00:21:04.524 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:04.534 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:04.534 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:04.561 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:11 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:04.639 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:04.642 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:05.174 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-23T00:21:05.451 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-23T00:21:06.069 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:06.116 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:06.117 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:06.117 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:06.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-02-23T00:21:06.389 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-23T00:21:06.839 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-23T00:21:06.969 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:13 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-23T00:21:07.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-23T00:21:07.522 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-23T00:21:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:07.911 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:08.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-02-23T00:21:08.535 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:08.535 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:08.535 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 24 k 2024-02-23T00:21:08.536 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 94 k 2024-02-23T00:21:08.537 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:21:08.811 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:03 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:09.059 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 46 kB/s | 24 kB 00:00 2024-02-23T00:21:09.060 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:09.061 INFO:teuthology.orchestra.run.smithi118.stdout:Total 46 kB/s | 24 kB 00:00 2024-02-23T00:21:09.061 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:21:09.065 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:21:09.065 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:21:09.068 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:21:09.071 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:21:09.358 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:21:09.592 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:09.874 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:10.226 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:10.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-02-23T00:21:10.435 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-02-23T00:21:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:10.437 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 968 k 2024-02-23T00:21:10.438 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 3.2 M 2024-02-23T00:21:10.438 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:10.584 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 12 MB/s | 885 kB 00:00 2024-02-23T00:21:10.709 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 418 kB/s | 83 kB 00:00 2024-02-23T00:21:10.710 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:10.710 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.5 MB/s | 968 kB 00:00 2024-02-23T00:21:10.719 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:10.739 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:10.740 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:10.824 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:10.827 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:10.998 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:11.106 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:18 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:11.275 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-23T00:21:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-23T00:21:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-23T00:21:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-23T00:21:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-23T00:21:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:12.227 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-23T00:21:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:12.228 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:12.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-02-23T00:21:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:12.675 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:12.676 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:12.676 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:12.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-02-23T00:21:13.236 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:08 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:13.572 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:20 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:14.781 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:14.782 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:21:14.783 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:14.783 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 131 k 2024-02-23T00:21:14.783 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 439 k 2024-02-23T00:21:14.783 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:14.854 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 131 kB 00:00 2024-02-23T00:21:14.855 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:14.855 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.8 MB/s | 131 kB 00:00 2024-02-23T00:21:14.855 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:14.860 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:14.860 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:14.909 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:14.912 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:15.098 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:15.146 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:15.147 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:15.147 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:15.248 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:15.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-02-23T00:21:15.563 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:15.935 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:15.936 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:16.007 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:23 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:16.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-02-23T00:21:16.921 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:11 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:17.515 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:17.561 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:17.562 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:17.562 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:17.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-02-23T00:21:18.415 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:25 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:18.429 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:18.476 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:18.477 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:18.477 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:18.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-02-23T00:21:19.686 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:14 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:19.906 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:19.952 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:19.953 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:19.953 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:20.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-02-23T00:21:20.806 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:27 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:21.245 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:21.245 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 24 k 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 94 k 2024-02-23T00:21:21.246 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:21.299 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 462 kB/s | 24 kB 00:00 2024-02-23T00:21:21.300 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:21.300 INFO:teuthology.orchestra.run.smithi090.stdout:Total 451 kB/s | 24 kB 00:00 2024-02-23T00:21:21.300 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:21.303 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:21.303 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:21.305 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:21.309 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:21.836 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:22.187 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:22.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-02-23T00:21:22.325 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:22.371 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:22.372 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:22.373 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:22.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-02-23T00:21:23.063 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:18 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:23.231 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:30 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:24.576 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:24.622 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:24.623 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:24.623 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:24.745 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:24.791 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:24.792 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-02-23T00:21:24.792 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:24.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-02-23T00:21:25.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-02-23T00:21:25.649 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:32 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:25.765 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:20 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:27.217 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:27.217 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:27.217 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:21:27.218 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:27.219 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 89 k 2024-02-23T00:21:27.219 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 298 k 2024-02-23T00:21:27.219 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:21:27.301 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:27.302 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 89 kB 00:00 2024-02-23T00:21:27.302 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:27.303 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.0 MB/s | 89 kB 00:00 2024-02-23T00:21:27.303 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:21:27.313 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:21:27.313 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:21:27.362 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:21:27.365 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:21:27.375 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:27.376 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:27.376 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:27.464 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:21:27.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-02-23T00:21:27.697 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:27.994 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:28.242 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:23 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:28.381 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:28.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-02-23T00:21:29.344 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:36 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:29.788 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:29.834 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:29.835 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:29.835 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:29.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-02-23T00:21:30.714 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:25 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:30.883 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:21:30.885 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:30.886 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:21:30.886 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:30.886 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.7 M 2024-02-23T00:21:30.886 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 18 M 2024-02-23T00:21:30.886 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:21:31.005 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2024-02-23T00:21:31.006 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:31.006 INFO:teuthology.orchestra.run.smithi118.stdout:Total 31 MB/s | 3.7 MB 00:00 2024-02-23T00:21:31.006 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:21:31.018 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:21:31.018 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:21:31.081 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:21:31.084 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:21:31.548 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:21:31.773 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:32.191 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:32.243 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:32.291 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:32.292 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:32.293 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:32.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:32.571 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:32.850 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-02-23T00:21:33.194 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:28 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:33.612 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:40 ago on Fri 23 Feb 2024 12:18:53 AM UTC. 2024-02-23T00:21:34.725 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:34.771 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:34.772 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:34.772 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:34.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-02-23T00:21:35.190 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-02-23T00:21:35.191 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:35.191 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:35.191 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:35.191 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-02-23T00:21:35.191 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 173 k 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 649 k 2024-02-23T00:21:35.192 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-02-23T00:21:35.234 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 173 kB 00:00 2024-02-23T00:21:35.234 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:35.234 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.9 MB/s | 173 kB 00:00 2024-02-23T00:21:35.235 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-02-23T00:21:35.245 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-02-23T00:21:35.245 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-02-23T00:21:35.295 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-02-23T00:21:35.298 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-02-23T00:21:35.413 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-02-23T00:21:35.629 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:35.655 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:30 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:36.312 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:36.313 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:36.313 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-02-23T00:21:36.313 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:36.313 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:21:36.313 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-02-23T00:21:36.551 DEBUG:teuthology.parallel:result is None 2024-02-23T00:21:37.178 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-23T00:21:37.226 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:37.227 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-02-23T00:21:37.227 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:37.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-02-23T00:21:38.133 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:33 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:39.701 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:39.702 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:39.703 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 89 k 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 298 k 2024-02-23T00:21:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:39.743 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-02-23T00:21:39.744 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:39.744 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-23T00:21:39.744 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:39.753 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:39.754 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:39.802 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:39.805 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:39.904 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:40.364 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:40.742 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:40.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-02-23T00:21:41.725 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:36 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:43.291 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:43.292 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:43.293 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:43.293 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:43.293 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:43.293 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-23T00:21:43.293 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.7 M 2024-02-23T00:21:43.294 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 18 M 2024-02-23T00:21:43.295 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:43.402 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-02-23T00:21:43.402 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:43.403 INFO:teuthology.orchestra.run.smithi090.stdout:Total 34 MB/s | 3.7 MB 00:00 2024-02-23T00:21:43.403 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:43.415 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:43.415 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:43.478 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:43.481 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:43.941 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:44.157 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:44.591 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:44.984 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:45.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-02-23T00:21:45.995 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:02:40 ago on Fri 23 Feb 2024 12:19:05 AM UTC. 2024-02-23T00:21:47.561 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:47.562 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 173 k 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 649 k 2024-02-23T00:21:47.563 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-02-23T00:21:47.628 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 173 kB 00:00 2024-02-23T00:21:47.628 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:21:47.628 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.6 MB/s | 173 kB 00:00 2024-02-23T00:21:47.628 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-02-23T00:21:47.638 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-02-23T00:21:47.638 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-02-23T00:21:47.687 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-02-23T00:21:47.690 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-02-23T00:21:47.805 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-02-23T00:21:47.997 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:48.638 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-23T00:21:48.638 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:48.638 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-02-23T00:21:48.638 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-23T00:21:48.639 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-02-23T00:21:48.639 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-02-23T00:21:48.851 DEBUG:teuthology.parallel:result is None 2024-02-23T00:21:48.851 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:21:48.852 INFO:teuthology.packaging:ref: None 2024-02-23T00:21:48.852 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:21:48.852 INFO:teuthology.packaging:branch: None 2024-02-23T00:21:48.852 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:21:48.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-23T00:21:49.009 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-23T00:21:49.032 INFO:teuthology.orchestra.run.smithi090.stdout:16.2.4-0.el8 2024-02-23T00:21:49.033 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-23T00:21:49.033 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-23T00:21:49.034 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-23T00:21:49.034 INFO:teuthology.packaging:ref: None 2024-02-23T00:21:49.035 INFO:teuthology.packaging:tag: v16.2.4 2024-02-23T00:21:49.035 INFO:teuthology.packaging:branch: None 2024-02-23T00:21:49.035 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:21:49.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-23T00:21:49.258 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-23T00:21:49.287 INFO:teuthology.orchestra.run.smithi118.stdout:16.2.4-0.el8 2024-02-23T00:21:49.287 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-23T00:21:49.288 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-23T00:21:49.291 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-23T00:21:49.292 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:21:49.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-23T00:21:49.321 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:21:49.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-23T00:21:49.360 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-23T00:21:49.361 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:21:49.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-02-23T00:21:49.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-23T00:21:49.454 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:21:49.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-02-23T00:21:49.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-23T00:21:49.561 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-23T00:21:49.561 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:21:49.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-23T00:21:49.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-23T00:21:49.655 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:21:49.656 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-23T00:21:49.688 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-23T00:21:49.759 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-23T00:21:49.759 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:21:49.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-02-23T00:21:49.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-23T00:21:49.853 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:21:49.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-02-23T00:21:49.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-23T00:21:49.954 INFO:teuthology.run_tasks:Running task print... 2024-02-23T00:21:49.978 INFO:teuthology.task.print:**** done install task... 2024-02-23T00:21:49.978 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-23T00:21:50.063 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(CEPHADM_STRAY_DAEMON\\)', '\\(FS_DEGRADED\\)', 'FS_', '\\(CEPHADM_', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'acting', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding', 'is down', 'osds down'], 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'} 2024-02-23T00:21:50.063 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-23T00:21:50.064 INFO:tasks.cephadm:Cluster fsid is 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:50.064 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-23T00:21:50.064 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-23T00:21:50.064 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi090': '172.21.15.90', 'mon.smithi118': '172.21.15.118'} 2024-02-23T00:21:50.064 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-23T00:21:50.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-02-23T00:21:50.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo hostname $(hostname -s) 2024-02-23T00:21:50.122 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-23T00:21:50.123 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.360 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 23 00:21 /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.361 DEBUG:teuthology.orchestra.run.smithi118:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.545 INFO:teuthology.orchestra.run.smithi118.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 23 00:21 /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.546 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.566 DEBUG:teuthology.orchestra.run.smithi118:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-23T00:21:50.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-02-23T00:21:50.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph 2024-02-23T00:21:50.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-02-23T00:21:50.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /etc/ceph 2024-02-23T00:21:50.753 INFO:tasks.cephadm:Writing seed config... 2024-02-23T00:21:50.754 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-23T00:21:50.754 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-23T00:21:50.754 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-23T00:21:50.754 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-23T00:21:50.754 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-23T00:21:50.755 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-23T00:21:50.756 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-23T00:21:50.757 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-23T00:21:50.757 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-23T00:21:50.757 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-23T00:21:50.757 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-23T00:21:50.757 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:21:50.758 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-23T00:21:50.775 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 898d23fe-d1e1-11ee-95c0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-23T00:21:50.775 DEBUG:teuthology.orchestra.run.smithi090:mon.smithi090> sudo journalctl -f -n 0 -u ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service 2024-02-23T00:21:50.817 INFO:tasks.cephadm:Bootstrapping... 2024-02-23T00:21:50.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.90 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:21:50.844 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:-- Logs begin at Fri 2024-02-23 00:03:04 UTC. -- 2024-02-23T00:21:51.087 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-23T00:21:52.307 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:21:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying podman|docker is present... 2024-02-23T00:21:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-23T00:21:52.341 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:21:52.356 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying lvm2 is present... 2024-02-23T00:21:52.356 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying time synchronization is in place... 2024-02-23T00:21:52.356 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled chrony.service 2024-02-23T00:21:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-23T00:21:52.363 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active chrony.service 2024-02-23T00:21:52.369 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:21:52.370 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-23T00:21:52.375 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: enabled 2024-02-23T00:21:52.376 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active chronyd.service 2024-02-23T00:21:52.381 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: active 2024-02-23T00:21:52.382 INFO:teuthology.orchestra.run.smithi090.stderr:Unit chronyd.service is enabled and running 2024-02-23T00:21:52.382 INFO:teuthology.orchestra.run.smithi090.stderr:Repeating the final host check... 2024-02-23T00:21:52.382 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-23T00:21:52.403 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:21:52.422 INFO:teuthology.orchestra.run.smithi090.stderr:podman|docker (/bin/podman) is present 2024-02-23T00:21:52.422 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl is present 2024-02-23T00:21:52.422 INFO:teuthology.orchestra.run.smithi090.stderr:lvcreate is present 2024-02-23T00:21:52.422 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled chrony.service 2024-02-23T00:21:52.428 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-23T00:21:52.428 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active chrony.service 2024-02-23T00:21:52.434 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:21:52.434 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-23T00:21:52.441 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: enabled 2024-02-23T00:21:52.441 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active chronyd.service 2024-02-23T00:21:52.445 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: active 2024-02-23T00:21:52.446 INFO:teuthology.orchestra.run.smithi090.stderr:Unit chronyd.service is enabled and running 2024-02-23T00:21:52.446 INFO:teuthology.orchestra.run.smithi090.stderr:Host looks OK 2024-02-23T00:21:52.446 INFO:teuthology.orchestra.run.smithi090.stderr:Cluster fsid: 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:52.447 INFO:teuthology.orchestra.run.smithi090.stderr:Acquiring lock 140060249237992 on /run/cephadm/898d23fe-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:21:52.447 INFO:teuthology.orchestra.run.smithi090.stderr:Lock 140060249237992 acquired on /run/cephadm/898d23fe-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:21:52.447 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying IP 172.21.15.90 port 3300 ... 2024-02-23T00:21:52.448 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying IP 172.21.15.90 port 6789 ... 2024-02-23T00:21:52.448 INFO:teuthology.orchestra.run.smithi090.stderr:Base mon IP is 172.21.15.90, final addrv is [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-02-23T00:21:52.448 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /sbin/ip route ls 2024-02-23T00:21:52.450 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.90 metric 100 2024-02-23T00:21:52.450 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-02-23T00:21:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /sbin/ip -6 route ls 2024-02-23T00:21:52.453 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-23T00:21:52.453 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-23T00:21:52.453 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-02-23T00:21:52.454 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /sbin/ip -6 addr ls 2024-02-23T00:21:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-23T00:21:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-23T00:21:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-23T00:21:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-23T00:21:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-02-23T00:21:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-23T00:21:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:Mon IP 172.21.15.90 is in CIDR network 172.21.0.0/20 2024-02-23T00:21:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-23T00:21:52.458 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-23T00:21:52.459 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-23T00:21:52.568 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-23T00:21:59.186 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Getting image source signatures 2024-02-23T00:21:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-23T00:21:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-23T00:22:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-23T00:22:11.856 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: Writing manifest to image destination 2024-02-23T00:22:12.183 INFO:teuthology.orchestra.run.smithi090.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-23T00:22:12.196 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-23T00:22:12.846 INFO:teuthology.orchestra.run.smithi090.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-23T00:22:13.414 INFO:teuthology.orchestra.run.smithi090.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-23T00:22:13.414 INFO:teuthology.orchestra.run.smithi090.stderr:Extracting ceph user uid/gid from container image... 2024-02-23T00:22:13.414 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-23T00:22:13.921 INFO:teuthology.orchestra.run.smithi090.stderr:stat: 167 167 2024-02-23T00:22:14.521 INFO:teuthology.orchestra.run.smithi090.stderr:Creating initial keys... 2024-02-23T00:22:14.521 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-23T00:22:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-authtool: AQA25ddlvljuNhAAZHRBpgJj0imjeWxDHtuZMQ== 2024-02-23T00:22:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-23T00:22:16.036 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-authtool: AQA35ddl5LTlOBAAuA8Xqr04FETq1kMcjgKooQ== 2024-02-23T00:22:16.591 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-23T00:22:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-authtool: AQA55ddlxvdGABAAgHylsCMWIW24CcA0UaqjXQ== 2024-02-23T00:22:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:Creating initial monmap... 2024-02-23T00:22:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpqgp9rr55:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 --addv smithi090 [v2:172.21.15.90:3300,v1:172.21.15.90:6789] /tmp/monmap 2024-02-23T00:22:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-23T00:22:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:18.160 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-23T00:22:18.743 INFO:teuthology.orchestra.run.smithi090.stderr:monmaptool for smithi090 [v2:172.21.15.90:3300,v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-23T00:22:18.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/monmaptool: set fsid to 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:18.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-23T00:22:18.743 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:22:18.743 INFO:teuthology.orchestra.run.smithi090.stderr:Creating mon... 2024-02-23T00:22:18.744 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/mon.smithi090:/var/lib/ceph/mon/ceph-smithi090:z -v /tmp/ceph-tmp2ggksyb_:/tmp/keyring:z -v /tmp/ceph-tmpqgp9rr55:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi090 --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-23T00:22:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.177+0000 7fcafe4fc700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.177+0000 7fcafe4fc700 1 imported monmap: 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: last_changed 2024-02-23T00:22:18.065834+0000 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: created 2024-02-23T00:22:18.065834+0000 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.smithi090 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.177+0000 7fcafe4fc700 0 /usr/bin/ceph-mon: set fsid to 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Compile date May 13 2021 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: DB SUMMARY 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi090/store.db dir, Total Num: 0, files: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi090/store.db: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:22:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.create_if_missing: 1 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.env: 0x55cbc4ec31c0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.info_log: 0x55cbc5d6aec0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:22:19.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.db_log_dir: 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi090/store.db 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:22:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.write_buffer_manager: 0x55cbc5d71530 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:22:19.307 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.row_cache: None 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.wal_filter: None 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:22:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:22:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Compression algorithms supported: 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kZSTD supported: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:22:19.310 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.178+0000 7fcafe4fc700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000001 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.merge_operator: 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:22:19.311 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cbc5c83cb8) 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-23T00:22:19.312 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_cache: 0x55cbc5cb8d10 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-23T00:22:19.313 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:22:19.314 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.num_levels: 7 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:22:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:22:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:22:19.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:22:19.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.179+0000 7fcafe4fc700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.180+0000 7fcafe4fc700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:22:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.180+0000 7fcafe4fc700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.181+0000 7fcafe4fc700 4 rocksdb: DB pointer 0x55cbc5d7d800 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.181+0000 7fcae7395700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.181+0000 7fcae7395700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:22:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:22:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:22:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: 2024-02-23T00:22:19.324 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.182+0000 7fcafe4fc700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-23T00:22:19.324 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.182+0000 7fcafe4fc700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-23T00:22:19.324 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:22:19.182+0000 7fcafe4fc700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi090 for mon.smithi090 2024-02-23T00:22:19.821 INFO:teuthology.orchestra.run.smithi090.stderr:create mon.smithi090 on 2024-02-23T00:22:19.822 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:19.826 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl disable ceph.target 2024-02-23T00:22:19.833 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-23T00:22:19.915 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl enable ceph.target 2024-02-23T00:22:19.922 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-23T00:22:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl start ceph.target 2024-02-23T00:22:20.009 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl enable ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target 2024-02-23T00:22:20.015 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target → /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target. 2024-02-23T00:22:20.015 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target → /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target. 2024-02-23T00:22:20.093 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target 2024-02-23T00:22:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl daemon-reload 2024-02-23T00:22:20.183 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:22:20.206 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl reset-failed ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:22:20.211 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Failed to reset failed state of unit ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service: Unit ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service not loaded. 2024-02-23T00:22:20.211 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl enable ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:22:20.218 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Created symlink /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target.wants/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service → /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@.service. 2024-02-23T00:22:20.304 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:22:20.823 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-23T00:22:20.830 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: disabled 2024-02-23T00:22:20.832 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active firewalld.service 2024-02-23T00:22:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:22:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:firewalld.service is not enabled 2024-02-23T00:22:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to enable service . firewalld.service is not available 2024-02-23T00:22:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-23T00:22:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mon to start... 2024-02-23T00:22:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mon... 2024-02-23T00:22:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/mon.smithi090:/var/lib/ceph/mon/ceph-smithi090:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-23T00:22:21.047 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:20 smithi090 bash[37755]: e1547c08bad5d4ef83f5aff56ba886eed79b3c12545f0cbf44390a70929bdd0e 2024-02-23T00:22:21.047 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:20 smithi090 systemd[1]: Started Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:22:21.350 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.348+0000 7f9b3084b700 1 Processor -- start 2024-02-23T00:22:21.350 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.348+0000 7f9b3084b700 1 -- start start 2024-02-23T00:22:21.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.348+0000 7f9b3084b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:21.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.348+0000 7f9b3084b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0fee90 con 0x7f9b2c070440 2024-02-23T00:22:21.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58282/0 (socket says 172.21.15.90:58282) 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2b7fe700 1 -- 172.21.15.90:0/828491876 learned_addr learned my addr 172.21.15.90:0/828491876 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2b7fe700 1 -- 172.21.15.90:0/828491876 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0fefd0 con 0x7f9b2c070440 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2b7fe700 1 --2- 172.21.15.90:0/828491876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9b2000b5f0 tx=0x7f9b20006ee0).ready entity=mon.0 client_cookie=100b910e0e926a28 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2a7fc700 1 -- 172.21.15.90:0/828491876 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b2000d600 con 0x7f9b2c070440 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2a7fc700 1 -- 172.21.15.90:0/828491876 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2000d760 con 0x7f9b2c070440 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.349+0000 7f9b2a7fc700 1 -- 172.21.15.90:0/828491876 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b2000da30 con 0x7f9b2c070440 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 -- 172.21.15.90:0/828491876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 msgr2=0x7f9b2c0fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 --2- 172.21.15.90:0/828491876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9b2000b5f0 tx=0x7f9b20006ee0).stop 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 -- 172.21.15.90:0/828491876 shutdown_connections 2024-02-23T00:22:21.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 --2- 172.21.15.90:0/828491876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c0fe9a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 -- 172.21.15.90:0/828491876 >> 172.21.15.90:0/828491876 conn(0x7f9b2c0f6420 msgr2=0x7f9b2c0f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 -- 172.21.15.90:0/828491876 shutdown_connections 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.350+0000 7f9b3084b700 1 -- 172.21.15.90:0/828491876 wait complete. 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b3084b700 1 Processor -- start 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b3084b700 1 -- start start 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b3084b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b3084b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c10a6c0 con 0x7f9b2c070440 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b2b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58294/0 (socket says 172.21.15.90:58294) 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.351+0000 7f9b2b7fe700 1 -- 172.21.15.90:0/312119995 learned_addr learned my addr 172.21.15.90:0/312119995 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b2b7fe700 1 -- 172.21.15.90:0/312119995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2000b200 con 0x7f9b2c070440 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b2b7fe700 1 --2- 172.21.15.90:0/312119995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9b20006c70 tx=0x7f9b20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b20011b40 con 0x7f9b2c070440 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20011ca0 con 0x7f9b2c070440 2024-02-23T00:22:21.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c10a8c0 con 0x7f9b2c070440 2024-02-23T00:22:21.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b2001d430 con 0x7f9b2c070440 2024-02-23T00:22:21.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c10ad50 con 0x7f9b2c070440 2024-02-23T00:22:21.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9b2000f070 con 0x7f9b2c070440 2024-02-23T00:22:21.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.352+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9b20027460 con 0x7f9b2c070440 2024-02-23T00:22:21.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.353+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c045ad0 con 0x7f9b2c070440 2024-02-23T00:22:21.358 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.355+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9b20016070 con 0x7f9b2c070440 2024-02-23T00:22:21.424 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.420+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9b2c103e70 con 0x7f9b2c070440 2024-02-23T00:22:21.424 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b13fff700 1 -- 172.21.15.90:0/312119995 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f9b20036070 con 0x7f9b2c070440 2024-02-23T00:22:21.424 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: cluster: 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: id: 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: services: 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi090 (age 0.645529s) 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: data: 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-23T00:22:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: pgs: 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 msgr2=0x7f9b2c10a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 --2- 172.21.15.90:0/312119995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9b20006c70 tx=0x7f9b20016000).stop 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 shutdown_connections 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 --2- 172.21.15.90:0/312119995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c070440 0x7f9b2c10a1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 >> 172.21.15.90:0/312119995 conn(0x7f9b2c0f6420 msgr2=0x7f9b2c0f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 shutdown_connections 2024-02-23T00:22:21.426 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:21.421+0000 7f9b3084b700 1 -- 172.21.15.90:0/312119995 wait complete. 2024-02-23T00:22:21.971 INFO:teuthology.orchestra.run.smithi090.stderr:mon is available 2024-02-23T00:22:21.972 INFO:teuthology.orchestra.run.smithi090.stderr:Assimilating anything we can from ceph.conf... 2024-02-23T00:22:21.972 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z -v /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/mon.smithi090:/var/lib/ceph/mon/ceph-smithi090:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi090/config 2024-02-23T00:22:22.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: cluster 2024-02-23T00:22:20.776512+0000 mon.smithi090 (mon.0) 2 : cluster [INF] mon.smithi090 is new leader, mons smithi090 in quorum (ranks 0) 2024-02-23T00:22:22.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: cluster 2024-02-23T00:22:20.776708+0000 mon.smithi090 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} 2024-02-23T00:22:22.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: cluster 2024-02-23T00:22:20.780152+0000 mon.smithi090 (mon.0) 4 : cluster [DBG] fsmap 2024-02-23T00:22:22.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: cluster 2024-02-23T00:22:20.782822+0000 mon.smithi090 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-23T00:22:22.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: cluster 2024-02-23T00:22:20.783372+0000 mon.smithi090 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-23T00:22:22.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: audit 2024-02-23T00:22:21.422013+0000 mon.smithi090 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.90:0/312119995' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.488+0000 7ffa2c6c4700 1 Processor -- start 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.488+0000 7ffa2c6c4700 1 -- start start 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.488+0000 7ffa2c6c4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.488+0000 7ffa2c6c4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa240fef40 con 0x7ffa24070430 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2a460700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2a460700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58306/0 (socket says 172.21.15.90:58306) 2024-02-23T00:22:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2a460700 1 -- 172.21.15.90:0/3041306374 learned_addr learned my addr 172.21.15.90:0/3041306374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2a460700 1 -- 172.21.15.90:0/3041306374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240ff080 con 0x7ffa24070430 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2a460700 1 --2- 172.21.15.90:0/3041306374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7ffa1400b5f0 tx=0x7ffa14006ee0).ready entity=mon.0 client_cookie=5b340605fc2033e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2945e700 1 -- 172.21.15.90:0/3041306374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1400d600 con 0x7ffa24070430 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2945e700 1 -- 172.21.15.90:0/3041306374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1400d760 con 0x7ffa24070430 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.489+0000 7ffa2945e700 1 -- 172.21.15.90:0/3041306374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1400da30 con 0x7ffa24070430 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/3041306374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 msgr2=0x7ffa240fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 --2- 172.21.15.90:0/3041306374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7ffa1400b5f0 tx=0x7ffa14006ee0).stop 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/3041306374 shutdown_connections 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 --2- 172.21.15.90:0/3041306374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa240fea50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/3041306374 >> 172.21.15.90:0/3041306374 conn(0x7ffa240f6530 msgr2=0x7ffa240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/3041306374 shutdown_connections 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/3041306374 wait complete. 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.490+0000 7ffa2c6c4700 1 Processor -- start 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2c6c4700 1 -- start start 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2c6c4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2c6c4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2410a850 con 0x7ffa24070430 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2a460700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2a460700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58310/0 (socket says 172.21.15.90:58310) 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2a460700 1 -- 172.21.15.90:0/2381275002 learned_addr learned my addr 172.21.15.90:0/2381275002 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2a460700 1 -- 172.21.15.90:0/2381275002 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1400b200 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2a460700 1 --2- 172.21.15.90:0/2381275002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ffa14006c70 tx=0x7ffa14016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa14011b40 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa14011ca0 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2410aa50 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1401d430 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.491+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2410aee0 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.492+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ffa1400f070 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.492+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ffa14027460 con 0x7ffa24070430 2024-02-23T00:22:22.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.493+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa24045ad0 con 0x7ffa24070430 2024-02-23T00:22:22.497 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.495+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ffa14016070 con 0x7ffa24070430 2024-02-23T00:22:22.560 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.559+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ffa24103f90 con 0x7ffa24070430 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.562+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ffa1401d5e0 con 0x7ffa24070430 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.562+0000 7ffa12ffd700 1 -- 172.21.15.90:0/2381275002 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7ffa14036020 con 0x7ffa24070430 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: [global] 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: fsid = 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.567 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: [mgr] 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: [osd] 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 msgr2=0x7ffa2410a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 --2- 172.21.15.90:0/2381275002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ffa14006c70 tx=0x7ffa14016000).stop 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 shutdown_connections 2024-02-23T00:22:22.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 --2- 172.21.15.90:0/2381275002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa24070430 0x7ffa2410a360 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 >> 172.21.15.90:0/2381275002 conn(0x7ffa240f6530 msgr2=0x7ffa240f81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 shutdown_connections 2024-02-23T00:22:22.569 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.563+0000 7ffa2c6c4700 1 -- 172.21.15.90:0/2381275002 wait complete. 2024-02-23T00:22:23.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: audit 2024-02-23T00:22:22.560447+0000 mon.smithi090 (mon.0) 8 : audit [INF] from='client.? 172.21.15.90:0/2381275002' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-23T00:22:23.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: audit 2024-02-23T00:22:22.563424+0000 mon.smithi090 (mon.0) 9 : audit [INF] from='client.? 172.21.15.90:0/2381275002' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-23T00:22:23.164 INFO:teuthology.orchestra.run.smithi090.stderr:Generating new minimal ceph.conf... 2024-02-23T00:22:23.164 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z -v /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/mon.smithi090:/var/lib/ceph/mon/ceph-smithi090:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi090/config 2024-02-23T00:22:23.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.693+0000 7f2f1ad9e700 1 Processor -- start 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.693+0000 7f2f1ad9e700 1 -- start start 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f1ad9e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f1ad9e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140728b0 con 0x7f2f14072490 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f19d9c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f19d9c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38330/0 (socket says 172.21.15.90:38330) 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f19d9c700 1 -- 172.21.15.90:0/125144562 learned_addr learned my addr 172.21.15.90:0/125144562 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f19d9c700 1 -- 172.21.15.90:0/125144562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14070f20 con 0x7f2f14072490 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f19d9c700 1 --2- 172.21.15.90:0/125144562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2f0400b5f0 tx=0x7f2f04006ee0).ready entity=mon.0 client_cookie=9eb7d5f473a26486 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f18d9a700 1 -- 172.21.15.90:0/125144562 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0400d600 con 0x7f2f14072490 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f18d9a700 1 -- 172.21.15.90:0/125144562 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2f0400dbf0 con 0x7f2f14072490 2024-02-23T00:22:23.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.694+0000 7f2f18d9a700 1 -- 172.21.15.90:0/125144562 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f04011cb0 con 0x7f2f14072490 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/125144562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 msgr2=0x7f2f14070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 --2- 172.21.15.90:0/125144562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2f0400b5f0 tx=0x7f2f04006ee0).stop 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/125144562 shutdown_connections 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 --2- 172.21.15.90:0/125144562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f14070a30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/125144562 >> 172.21.15.90:0/125144562 conn(0x7f2f140f7420 msgr2=0x7f2f140f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/125144562 shutdown_connections 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.695+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/125144562 wait complete. 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f1ad9e700 1 Processor -- start 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f1ad9e700 1 -- start start 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f1ad9e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f1ad9e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1410d060 con 0x7f2f14072490 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f19d9c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:23.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f19d9c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38344/0 (socket says 172.21.15.90:38344) 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f19d9c700 1 -- 172.21.15.90:0/3392499476 learned_addr learned my addr 172.21.15.90:0/3392499476 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f19d9c700 1 -- 172.21.15.90:0/3392499476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0400b200 con 0x7f2f14072490 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f19d9c700 1 --2- 172.21.15.90:0/3392499476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2f04006a20 tx=0x7f2f04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0401b570 con 0x7f2f14072490 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2f0401bb60 con 0x7f2f14072490 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0401aa20 con 0x7f2f14072490 2024-02-23T00:22:23.700 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.696+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1410d260 con 0x7f2f14072490 2024-02-23T00:22:23.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.697+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1410d6f0 con 0x7f2f14072490 2024-02-23T00:22:23.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.697+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2f0400f070 con 0x7f2f14072490 2024-02-23T00:22:23.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.697+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2f04025d00 con 0x7f2f14072490 2024-02-23T00:22:23.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.698+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14045ad0 con 0x7f2f14072490 2024-02-23T00:22:23.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.700+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2f04016070 con 0x7f2f14072490 2024-02-23T00:22:23.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.764+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2f14101e90 con 0x7f2f14072490 2024-02-23T00:22:23.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.764+0000 7f2f027fc700 1 -- 172.21.15.90:0/3392499476 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2f0401a370 con 0x7f2f14072490 2024-02-23T00:22:23.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 msgr2=0x7f2f1410cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:23.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 --2- 172.21.15.90:0/3392499476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2f04006a20 tx=0x7f2f04006ee0).stop 2024-02-23T00:22:23.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 shutdown_connections 2024-02-23T00:22:23.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 --2- 172.21.15.90:0/3392499476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f14072490 0x7f2f1410cb70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:23.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 >> 172.21.15.90:0/3392499476 conn(0x7f2f140f7420 msgr2=0x7f2f140f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:23.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 shutdown_connections 2024-02-23T00:22:23.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.765+0000 7f2f1ad9e700 1 -- 172.21.15.90:0/3392499476 wait complete. 2024-02-23T00:22:24.052 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: audit 2024-02-23T00:22:23.765798+0000 mon.smithi090 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.90:0/3392499476' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:22:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:Restarting the monitor... 2024-02-23T00:22:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl restart ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:22:24.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 systemd[1]: Stopping Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:22:24.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: debug 2024-02-23T00:22:24.407+0000 7fe78d750700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi090 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:22:24.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[37771]: debug 2024-02-23T00:22:24.407+0000 7fe78d750700 -1 mon.smithi090@0(leader) e1 *** Got Signal Terminated *** 2024-02-23T00:22:24.924 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 podman[38094]: ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090 2024-02-23T00:22:25.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 systemd[1]: ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service: Succeeded. 2024-02-23T00:22:25.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 systemd[1]: Stopped Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:22:25.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:24 smithi090 systemd[1]: Starting Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:22:25.439 INFO:teuthology.orchestra.run.smithi090.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-23T00:22:25.440 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.394+0000 7fa3df5b6700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.394+0000 7fa3df5b6700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.394+0000 7fa3df5b6700 0 pidfile_write: ignore empty --pid-file 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 0 load: jerasure load: lrc load: isa 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Compile date May 13 2021 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: DB SUMMARY 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: CURRENT file: CURRENT 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi090/store.db dir, Total Num: 1, files: 000004.sst 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi090/store.db: 000006.log size: 98702 ; 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.657 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.create_if_missing: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.env: 0x559905d501c0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.info_log: 0x5599082c5960 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:22:25.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.db_log_dir: 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi090/store.db 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:22:25.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.write_buffer_manager: 0x559908f3a030 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:22:25.660 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:22:25.661 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:22:25.662 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:22:25.662 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:22:25.662 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.row_cache: None 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.wal_filter: None 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:22:25.663 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:22:25.664 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Compression algorithms supported: 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kZSTD supported: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:22:25.665 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000005 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.merge_operator: 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5599081ddcb0) 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cache_index_and_filter_blocks: 1 2024-02-23T00:22:25.666 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: pin_top_level_index_and_filter: 1 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: index_type: 0 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: data_block_index_type: 0 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: index_shortening: 1 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: hash_index_allow_collision: 1 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: checksum: 1 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: no_block_cache: 0 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_cache: 0x559908212f10 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_cache_name: BinnedLRUCache 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_cache_options: 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: capacity : 536870912 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: num_shard_bits : 4 2024-02-23T00:22:25.667 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: strict_capacity_limit : 0 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: high_pri_pool_ratio: 0.000 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_cache_compressed: (nil) 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: persistent_cache: (nil) 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_size: 4096 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_size_deviation: 10 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_restart_interval: 16 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: index_block_restart_interval: 1 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: metadata_block_size: 4096 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: partition_filters: 0 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: use_delta_encoding: 1 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: whole_key_filtering: 1 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: verify_compression: 0 2024-02-23T00:22:25.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: read_amp_bytes_per_bit: 0 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: format_version: 2 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: enable_index_compression: 1 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: block_align: 0 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.num_levels: 7 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:22:25.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:22:25.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:22:25.671 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:22:25.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:22:25.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.399+0000 7fa3df5b6700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.400+0000 7fa3df5b6700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.400+0000 7fa3df5b6700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.400+0000 7fa3df5b6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647745401505, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.400+0000 7fa3df5b6700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.400+0000 7fa3df5b6700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.401+0000 7fa3df5b6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647745402336, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94276, "table_properties": {"data_size": 92364, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85395, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708647745, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.401+0000 7fa3df5b6700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-23T00:22:25.674 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.401+0000 7fa3df5b6700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647745402764, "job": 1, "event": "recovery_finished"} 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.402+0000 7fa3df5b6700 4 rocksdb: DB pointer 0x5599082d7800 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.402+0000 7fa3c63ce700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.402+0000 7fa3c63ce700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** DB Stats ** 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** Compaction Stats [default] ** 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** Compaction Stats [default] ** 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:25.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Cumulative compaction: 0.00 GB write, 32.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Interval compaction: 0.00 GB write, 32.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** Compaction Stats [default] ** 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:25.677 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** Compaction Stats [default] ** 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.7 0.00 0.00 1 0.001 0 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:22:25.678 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Cumulative compaction: 0.00 GB write, 31.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.403+0000 7fa3df5b6700 0 starting mon.smithi090 rank 0 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi090 fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 1 mon.smithi090@-1(???) e1 preinit fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).mds e1 new map 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).mds e1 print_map 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: e1 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: legacy client fscid: -1 2024-02-23T00:22:25.679 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: No filesystems configured 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 0 mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:23.765798+0000 mon.smithi090 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.90:0/3392499476' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:22:25.404+0000 7fa3df5b6700 1 mon.smithi090@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 bash[38165]: 5a3a59fbfe3b979c6abab7587b521fe08e9f78b14f0f06d58074995bd6b4f712 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 systemd[1]: Started Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:25.420642+0000 mon.smithi090 (mon.0) 1 : cluster [INF] mon.smithi090 is new leader, mons smithi090 in quorum (ranks 0) 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:25.420699+0000 mon.smithi090 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} 2024-02-23T00:22:25.680 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:25.420729+0000 mon.smithi090 (mon.0) 3 : cluster [DBG] fsmap 2024-02-23T00:22:25.681 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:25.420753+0000 mon.smithi090 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-23T00:22:25.681 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:25.421105+0000 mon.smithi090 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-23T00:22:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de7481700 1 Processor -- start 2024-02-23T00:22:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de7481700 1 -- start start 2024-02-23T00:22:25.959 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de7481700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de7481700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00fb6a0 con 0x7f8de00fad90 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de521d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de521d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38356/0 (socket says 172.21.15.90:38356) 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.954+0000 7f8de521d700 1 -- 172.21.15.90:0/2020800469 learned_addr learned my addr 172.21.15.90:0/2020800469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.955+0000 7f8de521d700 1 -- 172.21.15.90:0/2020800469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00fb7e0 con 0x7f8de00fad90 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.955+0000 7f8de521d700 1 --2- 172.21.15.90:0/2020800469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8ddc00b870 tx=0x7f8ddc006f70).ready entity=mon.0 client_cookie=e716f00009751f2d server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.955+0000 7f8dcffff700 1 -- 172.21.15.90:0/2020800469 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc00da20 con 0x7f8de00fad90 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.955+0000 7f8dcffff700 1 -- 172.21.15.90:0/2020800469 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8ddc010470 con 0x7f8de00fad90 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.955+0000 7f8dcffff700 1 -- 172.21.15.90:0/2020800469 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc00f5b0 con 0x7f8de00fad90 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- 172.21.15.90:0/2020800469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 msgr2=0x7f8de00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 --2- 172.21.15.90:0/2020800469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8ddc00b870 tx=0x7f8ddc006f70).stop 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- 172.21.15.90:0/2020800469 shutdown_connections 2024-02-23T00:22:25.960 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 --2- 172.21.15.90:0/2020800469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de00fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- 172.21.15.90:0/2020800469 >> 172.21.15.90:0/2020800469 conn(0x7f8de00f6530 msgr2=0x7f8de00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- 172.21.15.90:0/2020800469 shutdown_connections 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- 172.21.15.90:0/2020800469 wait complete. 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 Processor -- start 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 -- start start 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.956+0000 7f8de7481700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de7481700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0108df0 con 0x7f8de00fad90 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de521d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de521d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38368/0 (socket says 172.21.15.90:38368) 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de521d700 1 -- 172.21.15.90:0/2536998532 learned_addr learned my addr 172.21.15.90:0/2536998532 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de521d700 1 -- 172.21.15.90:0/2536998532 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc00b480 con 0x7f8de00fad90 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de521d700 1 --2- 172.21.15.90:0/2536998532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8ddc006800 tx=0x7f8ddc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc00f3d0 con 0x7f8de00fad90 2024-02-23T00:22:25.961 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8ddc00fd00 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ddc019d00 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.957+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de0108ff0 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.958+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de0109480 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.958+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8ddc016030 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.958+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8ddc024b30 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.959+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de0045ad0 con 0x7f8de00fad90 2024-02-23T00:22:25.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.961+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8ddc014070 con 0x7f8de00fad90 2024-02-23T00:22:26.025 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.024+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8de00fcae0 con 0x7f8de00fad90 2024-02-23T00:22:26.027 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.026+0000 7f8dcdffb700 1 -- 172.21.15.90:0/2536998532 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8ddc032030 con 0x7f8de00fad90 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 msgr2=0x7f8de0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 --2- 172.21.15.90:0/2536998532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8ddc006800 tx=0x7f8ddc006f70).stop 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 shutdown_connections 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 --2- 172.21.15.90:0/2536998532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de00fad90 0x7f8de0108900 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 >> 172.21.15.90:0/2536998532 conn(0x7f8de00f6530 msgr2=0x7f8de00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 shutdown_connections 2024-02-23T00:22:26.029 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:26.027+0000 7f8de7481700 1 -- 172.21.15.90:0/2536998532 wait complete. 2024-02-23T00:22:26.556 INFO:teuthology.orchestra.run.smithi090.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-23T00:22:26.556 INFO:teuthology.orchestra.run.smithi090.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:22:26.556 INFO:teuthology.orchestra.run.smithi090.stderr:Creating mgr... 2024-02-23T00:22:26.556 INFO:teuthology.orchestra.run.smithi090.stderr:Verifying port 9283 ... 2024-02-23T00:22:26.558 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl daemon-reload 2024-02-23T00:22:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz 2024-02-23T00:22:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl reset-failed ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz 2024-02-23T00:22:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Failed to reset failed state of unit ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz.service: Unit ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz.service not loaded. 2024-02-23T00:22:26.680 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl enable ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz 2024-02-23T00:22:26.686 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: Created symlink /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715.target.wants/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz.service → /etc/systemd/system/ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@.service. 2024-02-23T00:22:26.768 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mgr.smithi090.pxxmlz 2024-02-23T00:22:27.340 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:26.027375+0000 mon.smithi090 (mon.0) 6 : audit [INF] from='client.? 172.21.15.90:0/2536998532' entity='client.admin' 2024-02-23T00:22:27.344 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-23T00:22:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: disabled 2024-02-23T00:22:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active firewalld.service 2024-02-23T00:22:27.363 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:22:27.364 INFO:teuthology.orchestra.run.smithi090.stderr:firewalld.service is not enabled 2024-02-23T00:22:27.364 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to enable service . firewalld.service is not available 2024-02-23T00:22:27.364 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-23T00:22:27.365 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-23T00:22:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: disabled 2024-02-23T00:22:27.374 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active firewalld.service 2024-02-23T00:22:27.382 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:22:27.383 INFO:teuthology.orchestra.run.smithi090.stderr:firewalld.service is not enabled 2024-02-23T00:22:27.383 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-23T00:22:27.383 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mgr to start... 2024-02-23T00:22:27.383 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mgr... 2024-02-23T00:22:27.383 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-23T00:22:27.905 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.903+0000 7f99497ee700 1 Processor -- start 2024-02-23T00:22:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.903+0000 7f99497ee700 1 -- start start 2024-02-23T00:22:27.908 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.903+0000 7f99497ee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.903+0000 7f99497ee700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440fefa0 con 0x7f99440703d0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.904+0000 7f9942ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.904+0000 7f9942ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38384/0 (socket says 172.21.15.90:38384) 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.904+0000 7f9942ffd700 1 -- 172.21.15.90:0/68306010 learned_addr learned my addr 172.21.15.90:0/68306010 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.904+0000 7f9942ffd700 1 -- 172.21.15.90:0/68306010 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440ff0e0 con 0x7f99440703d0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.904+0000 7f9942ffd700 1 --2- 172.21.15.90:0/68306010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f993400b5f0 tx=0x7f9934006ee0).ready entity=mon.0 client_cookie=a4bca15e251c7557 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f9941ffb700 1 -- 172.21.15.90:0/68306010 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993400d600 con 0x7f99440703d0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f9941ffb700 1 -- 172.21.15.90:0/68306010 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f993400dbf0 con 0x7f99440703d0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f9941ffb700 1 -- 172.21.15.90:0/68306010 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9934011cb0 con 0x7f99440703d0 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 -- 172.21.15.90:0/68306010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 msgr2=0x7f99440feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 --2- 172.21.15.90:0/68306010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f993400b5f0 tx=0x7f9934006ee0).stop 2024-02-23T00:22:27.909 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 -- 172.21.15.90:0/68306010 shutdown_connections 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 --2- 172.21.15.90:0/68306010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440feab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 -- 172.21.15.90:0/68306010 >> 172.21.15.90:0/68306010 conn(0x7f99440f6530 msgr2=0x7f99440f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 -- 172.21.15.90:0/68306010 shutdown_connections 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.905+0000 7f99497ee700 1 -- 172.21.15.90:0/68306010 wait complete. 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.906+0000 7f99497ee700 1 Processor -- start 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.906+0000 7f99497ee700 1 -- start start 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.906+0000 7f99497ee700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.906+0000 7f99497ee700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99440fa9b0 con 0x7f99440703d0 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f9942ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f9942ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38396/0 (socket says 172.21.15.90:38396) 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f9942ffd700 1 -- 172.21.15.90:0/2395373383 learned_addr learned my addr 172.21.15.90:0/2395373383 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:27.910 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f9942ffd700 1 -- 172.21.15.90:0/2395373383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993400b200 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f9942ffd700 1 --2- 172.21.15.90:0/2395373383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9934006c70 tx=0x7f9934006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9934011400 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440fab50 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f993401b8a0 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993401a7c0 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99440fafe0 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.907+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f993400f070 con 0x7f99440703d0 2024-02-23T00:22:27.911 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.908+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9934025be0 con 0x7f99440703d0 2024-02-23T00:22:27.912 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.909+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9944045ad0 con 0x7f99440703d0 2024-02-23T00:22:27.914 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.913+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9934016070 con 0x7f99440703d0 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.980+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f99440008d0 con 0x7f99440703d0 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.980+0000 7f992b7fe700 1 -- 172.21.15.90:0/2395373383 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9934025e20 con 0x7f99440703d0 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsid": "898d23fe-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:22:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 0 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "smithi090" 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:22:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:22:27.990 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "available": false, 2024-02-23T00:22:27.991 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:22:20.778509+0000", 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:22:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 msgr2=0x7f99440fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 --2- 172.21.15.90:0/2395373383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9934006c70 tx=0x7f9934006ee0).stop 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 shutdown_connections 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 --2- 172.21.15.90:0/2395373383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99440703d0 0x7f99440fc530 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 >> 172.21.15.90:0/2395373383 conn(0x7f99440f6530 msgr2=0x7f99440f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 shutdown_connections 2024-02-23T00:22:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.981+0000 7f99497ee700 1 -- 172.21.15.90:0/2395373383 wait complete. 2024-02-23T00:22:28.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:27.981384+0000 mon.smithi090 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.90:0/2395373383' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:22:28.618 INFO:teuthology.orchestra.run.smithi090.stderr:mgr not available, waiting (1/15)... 2024-02-23T00:22:30.619 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-23T00:22:31.135 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.130+0000 7f9ab4e74700 1 Processor -- start 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.130+0000 7f9ab4e74700 1 -- start start 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.130+0000 7f9ab4e74700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.130+0000 7f9ab4e74700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00fb6a0 con 0x7f9ab00fad90 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38406/0 (socket says 172.21.15.90:38406) 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aae59c700 1 -- 172.21.15.90:0/2368001488 learned_addr learned my addr 172.21.15.90:0/2368001488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aae59c700 1 -- 172.21.15.90:0/2368001488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00fb7e0 con 0x7f9ab00fad90 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aae59c700 1 --2- 172.21.15.90:0/2368001488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9aa000b5f0 tx=0x7f9aa0006ee0).ready entity=mon.0 client_cookie=16695e5565bb5ddb server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aad59a700 1 -- 172.21.15.90:0/2368001488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa000d600 con 0x7f9ab00fad90 2024-02-23T00:22:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aad59a700 1 -- 172.21.15.90:0/2368001488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9aa000dbf0 con 0x7f9ab00fad90 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.131+0000 7f9aad59a700 1 -- 172.21.15.90:0/2368001488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa0011cb0 con 0x7f9ab00fad90 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- 172.21.15.90:0/2368001488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 msgr2=0x7f9ab00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 --2- 172.21.15.90:0/2368001488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9aa000b5f0 tx=0x7f9aa0006ee0).stop 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- 172.21.15.90:0/2368001488 shutdown_connections 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 --2- 172.21.15.90:0/2368001488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab00fb1b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- 172.21.15.90:0/2368001488 >> 172.21.15.90:0/2368001488 conn(0x7f9ab00f6530 msgr2=0x7f9ab00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- 172.21.15.90:0/2368001488 shutdown_connections 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- 172.21.15.90:0/2368001488 wait complete. 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 Processor -- start 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.132+0000 7f9ab4e74700 1 -- start start 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9ab4e74700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9ab4e74700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab010b020 con 0x7f9ab00fad90 2024-02-23T00:22:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9aae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9aae59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38410/0 (socket says 172.21.15.90:38410) 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9aae59c700 1 -- 172.21.15.90:0/3693851405 learned_addr learned my addr 172.21.15.90:0/3693851405 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9aae59c700 1 -- 172.21.15.90:0/3693851405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa000b200 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9aae59c700 1 --2- 172.21.15.90:0/3693851405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9aa0011ea0 tx=0x7f9aa0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa001b570 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9aa001bb60 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa001a8d0 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab010b220 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.133+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab010b6b0 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.134+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9aa000f070 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.134+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9aa0025de0 con 0x7f9ab00fad90 2024-02-23T00:22:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.134+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0045ad0 con 0x7f9ab00fad90 2024-02-23T00:22:31.139 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.136+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9aa0016070 con 0x7f9ab00fad90 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.204+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9ab00fcc30 con 0x7f9ab00fad90 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.205+0000 7f9a9effd700 1 -- 172.21.15.90:0/3693851405 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9aa0025460 con 0x7f9ab00fad90 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsid": "898d23fe-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:22:31.213 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 0 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "smithi090" 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:22:31.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:22:31.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "available": false, 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:22:20.778509+0000", 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:31.217 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 msgr2=0x7f9ab010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 --2- 172.21.15.90:0/3693851405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9aa0011ea0 tx=0x7f9aa0006ee0).stop 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 shutdown_connections 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 --2- 172.21.15.90:0/3693851405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ab00fad90 0x7f9ab010ab30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 >> 172.21.15.90:0/3693851405 conn(0x7f9ab00f6530 msgr2=0x7f9ab00700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 shutdown_connections 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.206+0000 7f9ab4e74700 1 -- 172.21.15.90:0/3693851405 wait complete. 2024-02-23T00:22:31.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:31.206141+0000 mon.smithi090 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.90:0/3693851405' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:22:31.859 INFO:teuthology.orchestra.run.smithi090.stderr:mgr not available, waiting (2/15)... 2024-02-23T00:22:32.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:32.167825+0000 mon.smithi090 (mon.0) 9 : cluster [INF] Activating manager daemon smithi090.pxxmlz 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:32.170473+0000 mon.smithi090 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi090.pxxmlz(active, starting, since 0.00274008s) 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.171535+0000 mon.smithi090 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.171654+0000 mon.smithi090 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.171751+0000 mon.smithi090 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.171853+0000 mon.smithi090 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.172212+0000 mon.smithi090 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pxxmlz", "id": "smithi090.pxxmlz"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:32.185431+0000 mon.smithi090 (mon.0) 16 : cluster [INF] Manager daemon smithi090.pxxmlz is now available 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.193435+0000 mon.smithi090 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.196950+0000 mon.smithi090 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.198064+0000 mon.smithi090 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.202544+0000 mon.smithi090 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:32.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:32.204517+0000 mon.smithi090 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.90:0/3138204565' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-23T00:22:34.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:33.173915+0000 mon.smithi090 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi090.pxxmlz(active, since 1.00618s) 2024-02-23T00:22:36.525 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:35.173955+0000 mon.smithi090 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi090.pxxmlz(active, since 3s) 2024-02-23T00:22:36.547 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.539+0000 7f6ff08bd700 1 Processor -- start 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.539+0000 7f6ff08bd700 1 -- start start 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.539+0000 7f6ff08bd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.539+0000 7f6ff08bd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0fb6a0 con 0x7f6fec0fad90 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fea59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fea59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60358/0 (socket says 172.21.15.90:60358) 2024-02-23T00:22:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fea59c700 1 -- 172.21.15.90:0/1664871211 learned_addr learned my addr 172.21.15.90:0/1664871211 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fea59c700 1 -- 172.21.15.90:0/1664871211 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0fb7e0 con 0x7f6fec0fad90 2024-02-23T00:22:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fea59c700 1 --2- 172.21.15.90:0/1664871211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6fe000b5f0 tx=0x7f6fe0006ee0).ready entity=mon.0 client_cookie=465fc09cd73f1185 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fe959a700 1 -- 172.21.15.90:0/1664871211 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe000d600 con 0x7f6fec0fad90 2024-02-23T00:22:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fe959a700 1 -- 172.21.15.90:0/1664871211 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6fe000dbf0 con 0x7f6fec0fad90 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.540+0000 7f6fe959a700 1 -- 172.21.15.90:0/1664871211 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe0011cb0 con 0x7f6fec0fad90 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 -- 172.21.15.90:0/1664871211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 msgr2=0x7f6fec0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/1664871211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6fe000b5f0 tx=0x7f6fe0006ee0).stop 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 -- 172.21.15.90:0/1664871211 shutdown_connections 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/1664871211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec0fb1b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 -- 172.21.15.90:0/1664871211 >> 172.21.15.90:0/1664871211 conn(0x7f6fec0f6530 msgr2=0x7f6fec0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 -- 172.21.15.90:0/1664871211 shutdown_connections 2024-02-23T00:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 -- 172.21.15.90:0/1664871211 wait complete. 2024-02-23T00:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.541+0000 7f6ff08bd700 1 Processor -- start 2024-02-23T00:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6ff08bd700 1 -- start start 2024-02-23T00:22:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6ff08bd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6ff08bd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec0704b0 con 0x7f6fec0fad90 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fea59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fea59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60374/0 (socket says 172.21.15.90:60374) 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fea59c700 1 -- 172.21.15.90:0/525783295 learned_addr learned my addr 172.21.15.90:0/525783295 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fea59c700 1 -- 172.21.15.90:0/525783295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe000b200 con 0x7f6fec0fad90 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fea59c700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6fe0011ea0 tx=0x7f6fe0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe001b570 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6fe001bb60 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0706b0 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.542+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe001a8d0 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.543+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec10df30 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.543+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f6fe000f070 con 0x7f6fec0fad90 2024-02-23T00:22:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.543+0000 7f6fdaffd700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 0x7f6fd00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.543+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6fe0041330 con 0x7f6fec0fad90 2024-02-23T00:22:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.543+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec045ad0 con 0x7f6fec0fad90 2024-02-23T00:22:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.544+0000 7f6fe9d9b700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 0x7f6fd00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.544+0000 7f6fe9d9b700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 0x7f6fd00322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6fdc0096a0 tx=0x7f6fdc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.548+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fe0016070 con 0x7f6fec0fad90 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.751+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6fec027a00 con 0x7f6fec0fad90 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.751+0000 7f6fdaffd700 1 -- 172.21.15.90:0/525783295 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6fe0022b70 con 0x7f6fec0fad90 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsid": "898d23fe-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:22:36.763 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 0 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "smithi090" 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:36.764 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:22:36.765 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:22:36.766 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "available": true, 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:22:36.767 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ], 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:22:20.778509+0000", 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: }, 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 msgr2=0x7f6fd00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:36.768 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 0x7f6fd00322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6fdc0096a0 tx=0x7f6fdc006b40).stop 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 msgr2=0x7f6fec10d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6fe0011ea0 tx=0x7f6fe0006ee0).stop 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 shutdown_connections 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f6fd002fe10 0x7f6fd00322d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 --2- 172.21.15.90:0/525783295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0fad90 0x7f6fec10d800 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 >> 172.21.15.90:0/525783295 conn(0x7f6fec0f6530 msgr2=0x7f6fec0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 shutdown_connections 2024-02-23T00:22:36.769 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:36.753+0000 7f6ff08bd700 1 -- 172.21.15.90:0/525783295 wait complete. 2024-02-23T00:22:37.369 INFO:teuthology.orchestra.run.smithi090.stderr:mgr is available 2024-02-23T00:22:37.369 INFO:teuthology.orchestra.run.smithi090.stderr:Enabling cephadm module... 2024-02-23T00:22:37.369 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-23T00:22:37.371 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:36.752318+0000 mon.smithi090 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.90:0/525783295' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:22:37.858 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2c742700 1 Processor -- start 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2c742700 1 -- start start 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2c742700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2c742700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240fef80 con 0x7fbf240703f0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2a4de700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.855+0000 7fbf2a4de700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60376/0 (socket says 172.21.15.90:60376) 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf2a4de700 1 -- 172.21.15.90:0/1616386584 learned_addr learned my addr 172.21.15.90:0/1616386584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf2a4de700 1 -- 172.21.15.90:0/1616386584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240ff0c0 con 0x7fbf240703f0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf2a4de700 1 --2- 172.21.15.90:0/1616386584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbf1400b5f0 tx=0x7fbf14006ee0).ready entity=mon.0 client_cookie=3dfa01705901bee4 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf294dc700 1 -- 172.21.15.90:0/1616386584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1400d600 con 0x7fbf240703f0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf294dc700 1 -- 172.21.15.90:0/1616386584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbf1400dbf0 con 0x7fbf240703f0 2024-02-23T00:22:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.856+0000 7fbf294dc700 1 -- 172.21.15.90:0/1616386584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf14011ce0 con 0x7fbf240703f0 2024-02-23T00:22:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 -- 172.21.15.90:0/1616386584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 msgr2=0x7fbf240fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 --2- 172.21.15.90:0/1616386584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbf1400b5f0 tx=0x7fbf14006ee0).stop 2024-02-23T00:22:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 -- 172.21.15.90:0/1616386584 shutdown_connections 2024-02-23T00:22:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 --2- 172.21.15.90:0/1616386584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fea90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 -- 172.21.15.90:0/1616386584 >> 172.21.15.90:0/1616386584 conn(0x7fbf240f6530 msgr2=0x7fbf240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 -- 172.21.15.90:0/1616386584 shutdown_connections 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.857+0000 7fbf2c742700 1 -- 172.21.15.90:0/1616386584 wait complete. 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2c742700 1 Processor -- start 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2c742700 1 -- start start 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2c742700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2c742700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240fa990 con 0x7fbf240703f0 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2a4de700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2a4de700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60386/0 (socket says 172.21.15.90:60386) 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2a4de700 1 -- 172.21.15.90:0/2231226757 learned_addr learned my addr 172.21.15.90:0/2231226757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2a4de700 1 -- 172.21.15.90:0/2231226757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1400b200 con 0x7fbf240703f0 2024-02-23T00:22:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf2a4de700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbf14011ed0 tx=0x7fbf14006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1401b570 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.858+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbf1401bb60 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf1401aad0 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf240fab30 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf240fafc0 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fbf1400f070 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf12ffd700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 0x7fbf0c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.859+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbf14041200 con 0x7fbf240703f0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.860+0000 7fbf29cdd700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 0x7fbf0c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.860+0000 7fbf29cdd700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 0x7fbf0c0322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbf200096a0 tx=0x7fbf20006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.860+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf24059100 con 0x7fbf240703f0 2024-02-23T00:22:37.865 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:37.864+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf14016070 con 0x7fbf240703f0 2024-02-23T00:22:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.049+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fbf24045ad0 con 0x7fbf240703f0 2024-02-23T00:22:38.176 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.175+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbf1401bcd0 con 0x7fbf240703f0 2024-02-23T00:22:38.177 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.176+0000 7fbf12ffd700 1 -- 172.21.15.90:0/2231226757 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fbf14005700 con 0x7fbf240703f0 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 msgr2=0x7fbf0c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 0x7fbf0c0322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbf200096a0 tx=0x7fbf20006b40).stop 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 msgr2=0x7fbf240fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbf14011ed0 tx=0x7fbf14006ee0).stop 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 shutdown_connections 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fbf0c02fe10 0x7fbf0c0322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 --2- 172.21.15.90:0/2231226757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf240703f0 0x7fbf240fc510 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:38.180 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 >> 172.21.15.90:0/2231226757 conn(0x7fbf240f6530 msgr2=0x7fbf240f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 shutdown_connections 2024-02-23T00:22:38.181 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:38.178+0000 7fbf2c742700 1 -- 172.21.15.90:0/2231226757 wait complete. 2024-02-23T00:22:38.514 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:38.050605+0000 mon.smithi090 (mon.0) 25 : audit [INF] from='client.? 172.21.15.90:0/2231226757' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-23T00:22:38.767 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-23T00:22:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.278+0000 7fc2db059700 1 Processor -- start 2024-02-23T00:22:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.278+0000 7fc2db059700 1 -- start start 2024-02-23T00:22:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.278+0000 7fc2db059700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.278+0000 7fc2db059700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40fef40 con 0x7fc2d4070430 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d8df5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d8df5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60426/0 (socket says 172.21.15.90:60426) 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d8df5700 1 -- 172.21.15.90:0/3782351947 learned_addr learned my addr 172.21.15.90:0/3782351947 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d8df5700 1 -- 172.21.15.90:0/3782351947 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40ff080 con 0x7fc2d4070430 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d8df5700 1 --2- 172.21.15.90:0/3782351947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc2c800b5f0 tx=0x7fc2c8006ee0).ready entity=mon.0 client_cookie=6902de8552b41567 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d37fe700 1 -- 172.21.15.90:0/3782351947 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c800d600 con 0x7fc2d4070430 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d37fe700 1 -- 172.21.15.90:0/3782351947 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc2c800dbf0 con 0x7fc2d4070430 2024-02-23T00:22:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.279+0000 7fc2d37fe700 1 -- 172.21.15.90:0/3782351947 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c8011cb0 con 0x7fc2d4070430 2024-02-23T00:22:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 -- 172.21.15.90:0/3782351947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 msgr2=0x7fc2d40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 --2- 172.21.15.90:0/3782351947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fc2c800b5f0 tx=0x7fc2c8006ee0).stop 2024-02-23T00:22:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 -- 172.21.15.90:0/3782351947 shutdown_connections 2024-02-23T00:22:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 --2- 172.21.15.90:0/3782351947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d40fea50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 -- 172.21.15.90:0/3782351947 >> 172.21.15.90:0/3782351947 conn(0x7fc2d40f6530 msgr2=0x7fc2d40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 -- 172.21.15.90:0/3782351947 shutdown_connections 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.280+0000 7fc2db059700 1 -- 172.21.15.90:0/3782351947 wait complete. 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.281+0000 7fc2db059700 1 Processor -- start 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.281+0000 7fc2db059700 1 -- start start 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.281+0000 7fc2db059700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.281+0000 7fc2db059700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d410b080 con 0x7fc2d4070430 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d8df5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d8df5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60436/0 (socket says 172.21.15.90:60436) 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d8df5700 1 -- 172.21.15.90:0/598522994 learned_addr learned my addr 172.21.15.90:0/598522994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d8df5700 1 -- 172.21.15.90:0/598522994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c800b200 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d8df5700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc2c800df90 tx=0x7fc2c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c801b410 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d410b280 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc2c801ba00 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2c801aa30 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.282+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d410b710 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc2c800f070 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2d17fa700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 0x7fc2b80322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc2c80420f0 con 0x7fc2d4070430 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2d3fff700 1 -- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 msgr2=0x7fc2b80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2d3fff700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 0x7fc2b80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:22:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.283+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d4045ad0 con 0x7fc2d4070430 2024-02-23T00:22:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.288+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2c8016070 con 0x7fc2d4070430 2024-02-23T00:22:39.485 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.484+0000 7fc2d3fff700 1 -- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 msgr2=0x7fc2b80322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:39.485 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.484+0000 7fc2d3fff700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 0x7fc2b80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.515+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc2d4104b20 con 0x7fc2d4070430 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.516+0000 7fc2d17fa700 1 -- 172.21.15.90:0/598522994 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc2c8022d50 con 0x7fc2d4070430 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "available": true, 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "active_name": "smithi090.pxxmlz", 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-23T00:22:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 msgr2=0x7fc2b80322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 0x7fc2b80322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 msgr2=0x7fc2d410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc2c800df90 tx=0x7fc2c8006ee0).stop 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 shutdown_connections 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7fc2b802fe10 0x7fc2b80322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 --2- 172.21.15.90:0/598522994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2d4070430 0x7fc2d410ab90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 >> 172.21.15.90:0/598522994 conn(0x7fc2d40f6530 msgr2=0x7fc2d40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 shutdown_connections 2024-02-23T00:22:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:39.518+0000 7fc2db059700 1 -- 172.21.15.90:0/598522994 wait complete. 2024-02-23T00:22:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:38.177109+0000 mon.smithi090 (mon.0) 26 : audit [INF] from='client.? 172.21.15.90:0/2231226757' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-23T00:22:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:38.177169+0000 mon.smithi090 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi090.pxxmlz(active, since 6s) 2024-02-23T00:22:40.067 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for the mgr to restart... 2024-02-23T00:22:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mgr epoch 5... 2024-02-23T00:22:40.068 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-23T00:22:40.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:39.516898+0000 mon.smithi090 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.90:0/598522994' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1723e1c700 1 Processor -- start 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1723e1c700 1 -- start start 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1723e1c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1723e1c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c0fb6a0 con 0x7f171c0fad90 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1721bb8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1721bb8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60450/0 (socket says 172.21.15.90:60450) 2024-02-23T00:22:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.607+0000 7f1721bb8700 1 -- 172.21.15.90:0/523544711 learned_addr learned my addr 172.21.15.90:0/523544711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.608+0000 7f1721bb8700 1 -- 172.21.15.90:0/523544711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c0fb7e0 con 0x7f171c0fad90 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.608+0000 7f1721bb8700 1 --2- 172.21.15.90:0/523544711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f171800b5f0 tx=0x7f1718006ee0).ready entity=mon.0 client_cookie=4cb480dc8a1f6a84 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.608+0000 7f1720bb6700 1 -- 172.21.15.90:0/523544711 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171800d600 con 0x7f171c0fad90 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.608+0000 7f1720bb6700 1 -- 172.21.15.90:0/523544711 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f171800dbf0 con 0x7f171c0fad90 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.608+0000 7f1720bb6700 1 -- 172.21.15.90:0/523544711 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1718011cb0 con 0x7f171c0fad90 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- 172.21.15.90:0/523544711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 msgr2=0x7f171c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 --2- 172.21.15.90:0/523544711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f171800b5f0 tx=0x7f1718006ee0).stop 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- 172.21.15.90:0/523544711 shutdown_connections 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 --2- 172.21.15.90:0/523544711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c0fb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- 172.21.15.90:0/523544711 >> 172.21.15.90:0/523544711 conn(0x7f171c0f6530 msgr2=0x7f171c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- 172.21.15.90:0/523544711 shutdown_connections 2024-02-23T00:22:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- 172.21.15.90:0/523544711 wait complete. 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 Processor -- start 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- start start 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.609+0000 7f1723e1c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c10b040 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1721bb8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1721bb8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60464/0 (socket says 172.21.15.90:60464) 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1721bb8700 1 -- 172.21.15.90:0/2610892087 learned_addr learned my addr 172.21.15.90:0/2610892087 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1721bb8700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171800b200 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1721bb8700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1718002f40 tx=0x7f1718006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171801b410 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f171801ba00 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c10b240 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171801a900 con 0x7f171c0fad90 2024-02-23T00:22:40.614 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.610+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c10b6d0 con 0x7f171c0fad90 2024-02-23T00:22:40.615 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f171800f070 con 0x7f171c0fad90 2024-02-23T00:22:40.615 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f170a7fc700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:40.616 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f171c059100 con 0x7f170402fe70 2024-02-23T00:22:40.616 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1718041280 con 0x7f171c0fad90 2024-02-23T00:22:40.616 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f17213b7700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 msgr2=0x7f1704032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:40.616 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.611+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:22:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.812+0000 7f17213b7700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 msgr2=0x7f1704032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:40.812+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:22:41.214 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:41.213+0000 7f17213b7700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 msgr2=0x7f1704032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:41.215 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:41.213+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-23T00:22:42.015 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:42.014+0000 7f17213b7700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 msgr2=0x7f1704032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/172496589 2024-02-23T00:22:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:42.014+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-23T00:22:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:43.142+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f1718025980 con 0x7f171c0fad90 2024-02-23T00:22:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:43.142+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 msgr2=0x7f1704032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:22:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:43.142+0000 7f170a7fc700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:43.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:43.090827+0000 mon.smithi090 (mon.0) 29 : cluster [INF] Active manager daemon smithi090.pxxmlz restarted 2024-02-23T00:22:43.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:43.091208+0000 mon.smithi090 (mon.0) 30 : cluster [INF] Activating manager daemon smithi090.pxxmlz 2024-02-23T00:22:43.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:43.093447+0000 mon.smithi090 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-23T00:22:44.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.149+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f1718024ca0 con 0x7f171c0fad90 2024-02-23T00:22:44.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.149+0000 7f170a7fc700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 0x7f1704035b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:44.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.149+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f171c059100 con 0x7f1704033790 2024-02-23T00:22:44.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.150+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 0x7f1704035b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:44.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.150+0000 7f17213b7700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 0x7f1704035b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f170c0036d0 tx=0x7f170c006d20).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:44.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.151+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f171c059100 con 0x7f1704033790 2024-02-23T00:22:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.157+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f171c045ad0 con 0x7f1704033790 2024-02-23T00:22:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.157+0000 7f170a7fc700 1 -- 172.21.15.90:0/2610892087 <== mgr.14114 v2:172.21.15.90:6800/3107336444 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f171c045ad0 con 0x7f1704033790 2024-02-23T00:22:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:22:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "initialized": true 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 msgr2=0x7f1704035b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 0x7f1704035b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f170c0036d0 tx=0x7f170c006d20).stop 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 msgr2=0x7f171c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1718002f40 tx=0x7f1718006ee0).stop 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 shutdown_connections 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f1704033790 0x7f1704035b80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f171c0fad90 0x7f171c10ab50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:44.162 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 --2- 172.21.15.90:0/2610892087 >> [v2:172.21.15.90:6800/172496589,v1:172.21.15.90:6801/172496589] conn(0x7f170402fe70 0x7f1704032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 >> 172.21.15.90:0/2610892087 conn(0x7f171c0f6530 msgr2=0x7f171c070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 shutdown_connections 2024-02-23T00:22:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:44.158+0000 7f1723e1c700 1 -- 172.21.15.90:0/2610892087 wait complete. 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:43.143812+0000 mon.smithi090 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi090.pxxmlz(active, starting, since 0.0526754s) 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.146259+0000 mon.smithi090 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.146856+0000 mon.smithi090 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pxxmlz", "id": "smithi090.pxxmlz"}]: dispatch 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.147591+0000 mon.smithi090 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.147809+0000 mon.smithi090 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.148030+0000 mon.smithi090 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:43.161234+0000 mon.smithi090 (mon.0) 38 : cluster [INF] Manager daemon smithi090.pxxmlz is now available 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.174941+0000 mon.smithi090 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.177553+0000 mon.smithi090 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.179915+0000 mon.smithi090 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.268364+0000 mon.smithi090 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.269035+0000 mon.smithi090 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:44.524 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.270602+0000 mon.smithi090 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:44.524 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.272012+0000 mon.smithi090 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:44.524 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.281758+0000 mon.smithi090 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:22:44.524 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:43.282856+0000 mon.smithi090 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:22:45.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:44.151784+0000 mgr.smithi090.pxxmlz (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-23T00:22:45.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:44.152766+0000 mon.smithi090 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi090.pxxmlz(active, since 1.06162s) 2024-02-23T00:22:45.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:44.158620+0000 mgr.smithi090.pxxmlz (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-23T00:22:46.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:22:45.153026+0000 mon.smithi090 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi090.pxxmlz(active, since 2s) 2024-02-23T00:22:49.194 INFO:teuthology.orchestra.run.smithi090.stderr:mgr epoch 5 is available 2024-02-23T00:22:49.195 INFO:teuthology.orchestra.run.smithi090.stderr:Setting orchestrator backend to cephadm... 2024-02-23T00:22:49.195 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-23T00:22:52.396 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f45650f2700 1 Processor -- start 2024-02-23T00:22:52.396 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f45650f2700 1 -- start start 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f45650f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f45650f2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600fef40 con 0x7f4560070430 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f455ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f455ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55698/0 (socket says 172.21.15.90:55698) 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.390+0000 7f455ed9d700 1 -- 172.21.15.90:0/2854398406 learned_addr learned my addr 172.21.15.90:0/2854398406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f455ed9d700 1 -- 172.21.15.90:0/2854398406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45600ff080 con 0x7f4560070430 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f455ed9d700 1 --2- 172.21.15.90:0/2854398406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f455000b5f0 tx=0x7f4550006ee0).ready entity=mon.0 client_cookie=9599694fbf3ec61f server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f455dd9b700 1 -- 172.21.15.90:0/2854398406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455000d600 con 0x7f4560070430 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f455dd9b700 1 -- 172.21.15.90:0/2854398406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f455000dbf0 con 0x7f4560070430 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f455dd9b700 1 -- 172.21.15.90:0/2854398406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4550011cb0 con 0x7f4560070430 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f45650f2700 1 -- 172.21.15.90:0/2854398406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 msgr2=0x7f45600fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f45650f2700 1 --2- 172.21.15.90:0/2854398406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f455000b5f0 tx=0x7f4550006ee0).stop 2024-02-23T00:22:52.397 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f45650f2700 1 -- 172.21.15.90:0/2854398406 shutdown_connections 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.391+0000 7f45650f2700 1 --2- 172.21.15.90:0/2854398406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f45600fea50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.392+0000 7f45650f2700 1 -- 172.21.15.90:0/2854398406 >> 172.21.15.90:0/2854398406 conn(0x7f45600f6530 msgr2=0x7f45600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.392+0000 7f45650f2700 1 -- 172.21.15.90:0/2854398406 shutdown_connections 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.392+0000 7f45650f2700 1 -- 172.21.15.90:0/2854398406 wait complete. 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.392+0000 7f45650f2700 1 Processor -- start 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f45650f2700 1 -- start start 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f45650f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f45650f2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456010b020 con 0x7f4560070430 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f455ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f455ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55714/0 (socket says 172.21.15.90:55714) 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f455ed9d700 1 -- 172.21.15.90:0/4089967300 learned_addr learned my addr 172.21.15.90:0/4089967300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f455ed9d700 1 -- 172.21.15.90:0/4089967300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455000b200 con 0x7f4560070430 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f455ed9d700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4550006800 tx=0x7f4550006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.393+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455001b570 con 0x7f4560070430 2024-02-23T00:22:52.398 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f455001bb60 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456010b220 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455001a9f0 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456010b6b0 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f455000f070 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.394+0000 7f45477fe700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 0x7f454c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.395+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f45500413b0 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.395+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560045ad0 con 0x7f4560070430 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.395+0000 7f455e59c700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 0x7f454c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:52.399 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.395+0000 7f455e59c700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 0x7f454c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:52.400 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.399+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4550016080 con 0x7f4560070430 2024-02-23T00:22:52.577 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.576+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f456010bfd0 con 0x7f454c02fe70 2024-02-23T00:22:52.582 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.581+0000 7f45477fe700 1 -- 172.21.15.90:0/4089967300 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f456010bfd0 con 0x7f454c02fe70 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 msgr2=0x7f454c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 0x7f454c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).stop 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 msgr2=0x7f456010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4550006800 tx=0x7f4550006ee0).stop 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 shutdown_connections 2024-02-23T00:22:52.585 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f454c02fe70 0x7f454c032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:52.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 --2- 172.21.15.90:0/4089967300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4560070430 0x7f456010ab30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:52.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 >> 172.21.15.90:0/4089967300 conn(0x7f45600f6530 msgr2=0x7f45600f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:52.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 shutdown_connections 2024-02-23T00:22:52.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:52.583+0000 7f45650f2700 1 -- 172.21.15.90:0/4089967300 wait complete. 2024-02-23T00:22:53.164 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-23T00:22:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.708+0000 7f269ec82700 1 Processor -- start 2024-02-23T00:22:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.708+0000 7f269ec82700 1 -- start start 2024-02-23T00:22:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ec82700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ec82700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980fefa0 con 0x7f26980703d0 2024-02-23T00:22:53.714 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ca1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ca1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38380/0 (socket says 172.21.15.90:38380) 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ca1e700 1 -- 172.21.15.90:0/1394551222 learned_addr learned my addr 172.21.15.90:0/1394551222 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ca1e700 1 -- 172.21.15.90:0/1394551222 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980ff0e0 con 0x7f26980703d0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f269ca1e700 1 --2- 172.21.15.90:0/1394551222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f268c00b5f0 tx=0x7f268c006ee0).ready entity=mon.0 client_cookie=a65c389747a0587b server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f26977fe700 1 -- 172.21.15.90:0/1394551222 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c00d600 con 0x7f26980703d0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f26977fe700 1 -- 172.21.15.90:0/1394551222 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f268c00dbf0 con 0x7f26980703d0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.709+0000 7f26977fe700 1 -- 172.21.15.90:0/1394551222 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c011cb0 con 0x7f26980703d0 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- 172.21.15.90:0/1394551222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 msgr2=0x7f26980feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 --2- 172.21.15.90:0/1394551222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f268c00b5f0 tx=0x7f268c006ee0).stop 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- 172.21.15.90:0/1394551222 shutdown_connections 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 --2- 172.21.15.90:0/1394551222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980feab0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- 172.21.15.90:0/1394551222 >> 172.21.15.90:0/1394551222 conn(0x7f26980f6530 msgr2=0x7f26980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:53.715 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- 172.21.15.90:0/1394551222 shutdown_connections 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- 172.21.15.90:0/1394551222 wait complete. 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 Processor -- start 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.710+0000 7f269ec82700 1 -- start start 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ec82700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ec82700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26980fa9b0 con 0x7f26980703d0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ca1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ca1e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38392/0 (socket says 172.21.15.90:38392) 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ca1e700 1 -- 172.21.15.90:0/225762306 learned_addr learned my addr 172.21.15.90:0/225762306 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ca1e700 1 -- 172.21.15.90:0/225762306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c00b200 con 0x7f26980703d0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ca1e700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f268c006a20 tx=0x7f268c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c01b570 con 0x7f26980703d0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f268c01bb60 con 0x7f26980703d0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.711+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26980fab50 con 0x7f26980703d0 2024-02-23T00:22:53.716 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c01aa20 con 0x7f26980703d0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26980fafe0 con 0x7f26980703d0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f268c00f070 con 0x7f26980703d0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f26957fa700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 0x7f267c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f268c041350 con 0x7f26980703d0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.712+0000 7f2697fff700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 0x7f267c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.713+0000 7f2697fff700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 0x7f267c0322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26840096a0 tx=0x7f2684006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:53.717 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.713+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698059100 con 0x7f26980703d0 2024-02-23T00:22:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.717+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f268c016070 con 0x7f26980703d0 2024-02-23T00:22:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:52.577835+0000 mgr.smithi090.pxxmlz (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:52.581775+0000 mon.smithi090 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:52.585413+0000 mon.smithi090 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:53.870 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.868+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f2698104b20 con 0x7f267c02fe10 2024-02-23T00:22:53.870 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.868+0000 7f26957fa700 1 -- 172.21.15.90:0/225762306 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f2698104b20 con 0x7f267c02fe10 2024-02-23T00:22:53.870 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: value unchanged 2024-02-23T00:22:53.872 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 msgr2=0x7f267c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:53.872 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 0x7f267c0322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26840096a0 tx=0x7f2684006b40).stop 2024-02-23T00:22:53.872 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 msgr2=0x7f26980fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:53.872 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f268c006a20 tx=0x7f268c006ee0).stop 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 shutdown_connections 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f267c02fe10 0x7f267c0322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 --2- 172.21.15.90:0/225762306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26980703d0 0x7f26980fc530 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 >> 172.21.15.90:0/225762306 conn(0x7f26980f6530 msgr2=0x7f26980f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 shutdown_connections 2024-02-23T00:22:53.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:53.870+0000 7f269ec82700 1 -- 172.21.15.90:0/225762306 wait complete. 2024-02-23T00:22:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:Generating ssh key... 2024-02-23T00:22:54.456 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-23T00:22:54.722 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:53.869694+0000 mgr.smithi090.pxxmlz (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:54.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcf4eec700 1 Processor -- start 2024-02-23T00:22:54.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcf4eec700 1 -- start start 2024-02-23T00:22:54.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcf4eec700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:54.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcf4eec700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00fb440 con 0x7efcf00fab30 2024-02-23T00:22:54.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcef7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcef7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38398/0 (socket says 172.21.15.90:38398) 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.961+0000 7efcef7fe700 1 -- 172.21.15.90:0/3004663331 learned_addr learned my addr 172.21.15.90:0/3004663331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcef7fe700 1 -- 172.21.15.90:0/3004663331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf00fb580 con 0x7efcf00fab30 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcef7fe700 1 --2- 172.21.15.90:0/3004663331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efcd800b5f0 tx=0x7efcd8006ee0).ready entity=mon.0 client_cookie=d4cb5d7913a15abd server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcee7fc700 1 -- 172.21.15.90:0/3004663331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd800d600 con 0x7efcf00fab30 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcee7fc700 1 -- 172.21.15.90:0/3004663331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efcd800dbf0 con 0x7efcf00fab30 2024-02-23T00:22:54.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcee7fc700 1 -- 172.21.15.90:0/3004663331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd8011cb0 con 0x7efcf00fab30 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 -- 172.21.15.90:0/3004663331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 msgr2=0x7efcf00faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 --2- 172.21.15.90:0/3004663331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efcd800b5f0 tx=0x7efcd8006ee0).stop 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 -- 172.21.15.90:0/3004663331 shutdown_connections 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 --2- 172.21.15.90:0/3004663331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf00faf50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 -- 172.21.15.90:0/3004663331 >> 172.21.15.90:0/3004663331 conn(0x7efcf00f6390 msgr2=0x7efcf00f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.962+0000 7efcf4eec700 1 -- 172.21.15.90:0/3004663331 shutdown_connections 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcf4eec700 1 -- 172.21.15.90:0/3004663331 wait complete. 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcf4eec700 1 Processor -- start 2024-02-23T00:22:54.970 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcf4eec700 1 -- start start 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcf4eec700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcf4eec700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00ff310 con 0x7efcf00fab30 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.963+0000 7efcef7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcef7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38414/0 (socket says 172.21.15.90:38414) 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcef7fe700 1 -- 172.21.15.90:0/2235618112 learned_addr learned my addr 172.21.15.90:0/2235618112 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcef7fe700 1 -- 172.21.15.90:0/2235618112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd800b200 con 0x7efcf00fab30 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcef7fe700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efcd8011ea0 tx=0x7efcd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:54.971 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd801b570 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf00ff4b0 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efcd801bb60 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd801a8d0 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.964+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf00ff940 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.965+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efcd800f070 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.965+0000 7efce7fff700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 0x7efcd00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.965+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0045ad0 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.965+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efcd8041450 con 0x7efcf00fab30 2024-02-23T00:22:54.972 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.965+0000 7efceeffd700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 0x7efcd00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:54.973 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.966+0000 7efceeffd700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 0x7efcd00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7efce00096a0 tx=0x7efce0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:54.973 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:54.969+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efcd8016080 con 0x7efcf00fab30 2024-02-23T00:22:55.120 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.119+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efcf00ffb80 con 0x7efcd002fe10 2024-02-23T00:22:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.270+0000 7efce7fff700 1 -- 172.21.15.90:0/2235618112 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efcf00ffb80 con 0x7efcd002fe10 2024-02-23T00:22:55.274 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 msgr2=0x7efcd00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:55.274 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 0x7efcd00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7efce00096a0 tx=0x7efce0006b40).stop 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 msgr2=0x7efcf0100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efcd8011ea0 tx=0x7efcd8006ee0).stop 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 shutdown_connections 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efcd002fe10 0x7efcd00322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 --2- 172.21.15.90:0/2235618112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efcf00fab30 0x7efcf0100e90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.272+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 >> 172.21.15.90:0/2235618112 conn(0x7efcf00f6390 msgr2=0x7efcf0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.273+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 shutdown_connections 2024-02-23T00:22:55.275 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:55.273+0000 7efcf4eec700 1 -- 172.21.15.90:0/2235618112 wait complete. 2024-02-23T00:22:55.889 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-23T00:22:55.892 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:22:55.120785+0000 mgr.smithi090.pxxmlz (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-23T00:22:55.892 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:55.268506+0000 mon.smithi090 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:55.892 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:55.270839+0000 mon.smithi090 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.427+0000 7fb08586e700 1 Processor -- start 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.427+0000 7fb08586e700 1 -- start start 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.427+0000 7fb08586e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.427+0000 7fb08586e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0800fdde0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:56.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38426/0 (socket says 172.21.15.90:38426) 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07effd700 1 -- 172.21.15.90:0/2372174155 learned_addr learned my addr 172.21.15.90:0/2372174155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07effd700 1 -- 172.21.15.90:0/2372174155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0800fdf20 con 0x7fb0800fd4d0 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07effd700 1 --2- 172.21.15.90:0/2372174155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb06800b5f0 tx=0x7fb068006ee0).ready entity=mon.0 client_cookie=2078f3cf47ba0b71 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07dffb700 1 -- 172.21.15.90:0/2372174155 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb06800d600 con 0x7fb0800fd4d0 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07dffb700 1 -- 172.21.15.90:0/2372174155 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb06800dbf0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.428+0000 7fb07dffb700 1 -- 172.21.15.90:0/2372174155 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb068011cb0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 -- 172.21.15.90:0/2372174155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 msgr2=0x7fb0800fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 --2- 172.21.15.90:0/2372174155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb06800b5f0 tx=0x7fb068006ee0).stop 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 -- 172.21.15.90:0/2372174155 shutdown_connections 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 --2- 172.21.15.90:0/2372174155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb0800fd8f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 -- 172.21.15.90:0/2372174155 >> 172.21.15.90:0/2372174155 conn(0x7fb0800f5d30 msgr2=0x7fb0800f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 -- 172.21.15.90:0/2372174155 shutdown_connections 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 -- 172.21.15.90:0/2372174155 wait complete. 2024-02-23T00:22:56.433 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.429+0000 7fb08586e700 1 Processor -- start 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb08586e700 1 -- start start 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb08586e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb08586e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08010b0a0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb07effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb07effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38444/0 (socket says 172.21.15.90:38444) 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb07effd700 1 -- 172.21.15.90:0/290881976 learned_addr learned my addr 172.21.15.90:0/290881976 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb07effd700 1 -- 172.21.15.90:0/290881976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06800b200 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb07effd700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb068002f40 tx=0x7fb068006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb06801b410 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08010b2a0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb06801ba00 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb06801a980 con 0x7fb0800fd4d0 2024-02-23T00:22:56.434 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.430+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08010b730 con 0x7fb0800fd4d0 2024-02-23T00:22:56.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.431+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb06800f070 con 0x7fb0800fd4d0 2024-02-23T00:22:56.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.431+0000 7fb0777fe700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 0x7fb0600322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:56.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.431+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb068041470 con 0x7fb0800fd4d0 2024-02-23T00:22:56.436 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.431+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb080045ad0 con 0x7fb0800fd4d0 2024-02-23T00:22:56.436 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.431+0000 7fb07e7fc700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 0x7fb0600322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:56.436 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.432+0000 7fb07e7fc700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 0x7fb0600322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb0700096a0 tx=0x7fb070006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:56.437 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.436+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb068016070 con 0x7fb0800fd4d0 2024-02-23T00:22:56.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.584+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb080104780 con 0x7fb06002fe10 2024-02-23T00:22:56.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.585+0000 7fb0777fe700 1 -- 172.21.15.90:0/290881976 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb080104780 con 0x7fb06002fe10 2024-02-23T00:22:56.586 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfoeZLitAzd3QiNOepmW0noMkEd0yXgN49QijoA4pdW89K55GvZWsT4fIM4cOSBD5papI5vrly5TsHlvwZ9qQsP/CKwbN1ayLehyVtU13zFocyUJqRwZ76Q27Prko20rgkVzWV30OrRUc5KJOGo0AXnUvK5DcHjKEAB1fx4m//Hmxo8qkDyOxqdNHq3RjLcn49AYFadKPcF1JFGX69KVX/a1KB+fwISwlMfdaJAskshhIqKAE+AyC/2YJlPbc0qd2GgCv4VYfm03gNUcgRKUUaT7IXWCDxkMhiG2eJAH9J2fv6PAkvPrt6oaqH4GA8TCLoqhFVrMnuKx8xrP9tNMbZ4GhXLNkoWTLDj69kKwlnQQDCJW/K5VgMKMklbNd4fhqXhqxx1M4WxVMkE2o8rgfA9IOr4XbkRSEK8FeSwScfza3qTi4pFhW/jbVvzn/XlRifaTg3YjNoVfy3TbUwGXSqupF7Fh3hXOG2cw00qVOxKG8/6usz410A0acHaHbZGeE= ceph-898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:56.588 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 msgr2=0x7fb0600322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:56.588 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 0x7fb0600322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb0700096a0 tx=0x7fb070006b40).stop 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 msgr2=0x7fb08010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb068002f40 tx=0x7fb068006ee0).stop 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 shutdown_connections 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fb06002fe10 0x7fb0600322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 --2- 172.21.15.90:0/290881976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0800fd4d0 0x7fb08010abb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 >> 172.21.15.90:0/290881976 conn(0x7fb0800f5d30 msgr2=0x7fb0800ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 shutdown_connections 2024-02-23T00:22:56.589 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:56.586+0000 7fb08586e700 1 -- 172.21.15.90:0/290881976 wait complete. 2024-02-23T00:22:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:22:57.316 INFO:teuthology.orchestra.run.smithi090.stderr:Adding key to root@localhost authorized_keys... 2024-02-23T00:22:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:Adding host smithi090... 2024-02-23T00:22:57.317 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi090 172.21.15.90 2024-02-23T00:22:57.521 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:56.585861+0000 mgr.smithi090.pxxmlz (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5f69a700 1 Processor -- start 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5f69a700 1 -- start start 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5f69a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5f69a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe580fef80 con 0x7fbe580703f0 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5d436700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:57.812 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5d436700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38446/0 (socket says 172.21.15.90:38446) 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.807+0000 7fbe5d436700 1 -- 172.21.15.90:0/3227045318 learned_addr learned my addr 172.21.15.90:0/3227045318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe5d436700 1 -- 172.21.15.90:0/3227045318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe580ff0c0 con 0x7fbe580703f0 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe5d436700 1 --2- 172.21.15.90:0/3227045318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbe5400b5f0 tx=0x7fbe54006ee0).ready entity=mon.0 client_cookie=fb0ad5e71c9b1f10 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe47fff700 1 -- 172.21.15.90:0/3227045318 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe5400d600 con 0x7fbe580703f0 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe47fff700 1 -- 172.21.15.90:0/3227045318 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbe5400dbf0 con 0x7fbe580703f0 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe47fff700 1 -- 172.21.15.90:0/3227045318 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe54011ce0 con 0x7fbe580703f0 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3227045318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 msgr2=0x7fbe580fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.808+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3227045318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fbe5400b5f0 tx=0x7fbe54006ee0).stop 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3227045318 shutdown_connections 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3227045318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe580fea90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3227045318 >> 172.21.15.90:0/3227045318 conn(0x7fbe580f6530 msgr2=0x7fbe580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3227045318 shutdown_connections 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3227045318 wait complete. 2024-02-23T00:22:57.813 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 Processor -- start 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- start start 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5f69a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5810b010 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5d436700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5d436700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38462/0 (socket says 172.21.15.90:38462) 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.809+0000 7fbe5d436700 1 -- 172.21.15.90:0/3329586040 learned_addr learned my addr 172.21.15.90:0/3329586040 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe5d436700 1 -- 172.21.15.90:0/3329586040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5400b200 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe5d436700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbe54011ed0 tx=0x7fbe54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe5401b570 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbe5401bb60 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe5401aad0 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5810b210 con 0x7fbe580703f0 2024-02-23T00:22:57.814 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.810+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5810b6a0 con 0x7fbe580703f0 2024-02-23T00:22:57.815 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.811+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbe5400f070 con 0x7fbe580703f0 2024-02-23T00:22:57.815 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.811+0000 7fbe45ffb700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 0x7fbe40032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:57.816 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.811+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe58045ad0 con 0x7fbe580703f0 2024-02-23T00:22:57.816 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.811+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbe54041370 con 0x7fbe580703f0 2024-02-23T00:22:57.816 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.811+0000 7fbe5cc35700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 0x7fbe40032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:57.816 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.812+0000 7fbe5cc35700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 0x7fbe40032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbe480096a0 tx=0x7fbe48006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:57.817 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.815+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe54016080 con 0x7fbe580703f0 2024-02-23T00:22:57.987 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:57.985+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe5810bfc0 con 0x7fbe4002fe70 2024-02-23T00:22:58.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:57.986807+0000 mgr.smithi090.pxxmlz (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:58.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Added host 'smithi090' 2024-02-23T00:22:58.568 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.565+0000 7fbe45ffb700 1 -- 172.21.15.90:0/3329586040 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fbe5810bfc0 con 0x7fbe4002fe70 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 msgr2=0x7fbe40032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 0x7fbe40032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbe480096a0 tx=0x7fbe48006b40).stop 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 msgr2=0x7fbe5810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbe54011ed0 tx=0x7fbe54006ee0).stop 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 shutdown_connections 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fbe4002fe70 0x7fbe40032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 --2- 172.21.15.90:0/3329586040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe580703f0 0x7fbe5810ab20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.567+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 >> 172.21.15.90:0/3329586040 conn(0x7fbe580f6530 msgr2=0x7fbe580f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.568+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 shutdown_connections 2024-02-23T00:22:58.571 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:58.568+0000 7fbe5f69a700 1 -- 172.21.15.90:0/3329586040 wait complete. 2024-02-23T00:22:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying mon service with default placement... 2024-02-23T00:22:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-23T00:22:59.693 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f6425f3a700 1 Processor -- start 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f6425f3a700 1 -- start start 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f6425f3a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f6425f3a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64200fc8c0 con 0x7f64200703b0 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38464/0 (socket says 172.21.15.90:38464) 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f641f7fe700 1 -- 172.21.15.90:0/1262692037 learned_addr learned my addr 172.21.15.90:0/1262692037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.691+0000 7f641f7fe700 1 -- 172.21.15.90:0/1262692037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64201013c0 con 0x7f64200703b0 2024-02-23T00:22:59.694 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f641f7fe700 1 --2- 172.21.15.90:0/1262692037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f641000b5f0 tx=0x7f6410006ee0).ready entity=mon.0 client_cookie=6a4eef7f9f612ea5 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f641e7fc700 1 -- 172.21.15.90:0/1262692037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641000d600 con 0x7f64200703b0 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f641e7fc700 1 -- 172.21.15.90:0/1262692037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f641000dbf0 con 0x7f64200703b0 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f641e7fc700 1 -- 172.21.15.90:0/1262692037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6410011cb0 con 0x7f64200703b0 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 -- 172.21.15.90:0/1262692037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 msgr2=0x7f6420100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 --2- 172.21.15.90:0/1262692037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f641000b5f0 tx=0x7f6410006ee0).stop 2024-02-23T00:22:59.696 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 -- 172.21.15.90:0/1262692037 shutdown_connections 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 --2- 172.21.15.90:0/1262692037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f6420100ed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 -- 172.21.15.90:0/1262692037 >> 172.21.15.90:0/1262692037 conn(0x7f64200f6530 msgr2=0x7f64200f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 -- 172.21.15.90:0/1262692037 shutdown_connections 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.692+0000 7f6425f3a700 1 -- 172.21.15.90:0/1262692037 wait complete. 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.693+0000 7f6425f3a700 1 Processor -- start 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.693+0000 7f6425f3a700 1 -- start start 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.693+0000 7f6425f3a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.693+0000 7f6425f3a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642010d270 con 0x7f64200703b0 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.693+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f641f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38470/0 (socket says 172.21.15.90:38470) 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f641f7fe700 1 -- 172.21.15.90:0/3495284403 learned_addr learned my addr 172.21.15.90:0/3495284403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:22:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f641f7fe700 1 -- 172.21.15.90:0/3495284403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641000b200 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f641f7fe700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6410006800 tx=0x7f6410006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641001b570 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642010d470 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f641001bb60 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f641001a9f0 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.694+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642010e080 con 0x7f64200703b0 2024-02-23T00:22:59.698 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.695+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6420059100 con 0x7f64200703b0 2024-02-23T00:22:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.695+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f641000f070 con 0x7f64200703b0 2024-02-23T00:22:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.696+0000 7f6407fff700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 0x7f640c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.696+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6410041c10 con 0x7f64200703b0 2024-02-23T00:22:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.696+0000 7f641effd700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 0x7f640c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.696+0000 7f641effd700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 0x7f640c0322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64140096a0 tx=0x7f6414006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.700+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6410023b10 con 0x7f64200703b0 2024-02-23T00:22:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.871+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f642010e380 con 0x7f640c02fe10 2024-02-23T00:22:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.875+0000 7f6407fff700 1 -- 172.21.15.90:0/3495284403 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f642010e380 con 0x7f640c02fe10 2024-02-23T00:22:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-23T00:22:59.879 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 msgr2=0x7f640c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:59.879 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 0x7f640c0322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64140096a0 tx=0x7f6414006b40).stop 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 msgr2=0x7f642010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6410006800 tx=0x7f6410006ee0).stop 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 shutdown_connections 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f640c02fe10 0x7f640c0322d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 --2- 172.21.15.90:0/3495284403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64200703b0 0x7f642010cd80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 >> 172.21.15.90:0/3495284403 conn(0x7f64200f6530 msgr2=0x7f64200f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 shutdown_connections 2024-02-23T00:22:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:22:59.877+0000 7f6425f3a700 1 -- 172.21.15.90:0/3495284403 wait complete. 2024-02-23T00:22:59.899 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:58.565457+0000 mon.smithi090 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:22:59.899 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:22:58.565745+0000 mgr.smithi090.pxxmlz (mgr.14114) 9 : cephadm [INF] Added host smithi090 2024-02-23T00:22:59.899 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:22:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:58.565975+0000 mon.smithi090 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:00.416 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying mgr service with default placement... 2024-02-23T00:23:00.416 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.956+0000 7f86f9eef700 1 Processor -- start 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.956+0000 7f86f9eef700 1 -- start start 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f9eef700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f9eef700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40fef40 con 0x7f86f4070430 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38484/0 (socket says 172.21.15.90:38484) 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f37fe700 1 -- 172.21.15.90:0/3153636314 learned_addr learned my addr 172.21.15.90:0/3153636314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:00.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f37fe700 1 -- 172.21.15.90:0/3153636314 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40ff080 con 0x7f86f4070430 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f37fe700 1 --2- 172.21.15.90:0/3153636314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f86e800b5f0 tx=0x7f86e8006ee0).ready entity=mon.0 client_cookie=8e71b22ddb0d68f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f27fc700 1 -- 172.21.15.90:0/3153636314 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e800d600 con 0x7f86f4070430 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f27fc700 1 -- 172.21.15.90:0/3153636314 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f86e800dbf0 con 0x7f86f4070430 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.957+0000 7f86f27fc700 1 -- 172.21.15.90:0/3153636314 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e8011cb0 con 0x7f86f4070430 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 -- 172.21.15.90:0/3153636314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 msgr2=0x7f86f40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3153636314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f86e800b5f0 tx=0x7f86e8006ee0).stop 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 -- 172.21.15.90:0/3153636314 shutdown_connections 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3153636314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f40fea50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 -- 172.21.15.90:0/3153636314 >> 172.21.15.90:0/3153636314 conn(0x7f86f40f6530 msgr2=0x7f86f40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 -- 172.21.15.90:0/3153636314 shutdown_connections 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 -- 172.21.15.90:0/3153636314 wait complete. 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.958+0000 7f86f9eef700 1 Processor -- start 2024-02-23T00:23:00.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f9eef700 1 -- start start 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f9eef700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f9eef700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f410b030 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38490/0 (socket says 172.21.15.90:38490) 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f37fe700 1 -- 172.21.15.90:0/3944261155 learned_addr learned my addr 172.21.15.90:0/3944261155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f37fe700 1 -- 172.21.15.90:0/3944261155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e800b200 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f37fe700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f86e8006800 tx=0x7f86e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e801b570 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f410b230 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f86e801bb60 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e801a9f0 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.959+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f410b6c0 con 0x7f86f4070430 2024-02-23T00:23:00.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.960+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f86e800f070 con 0x7f86f4070430 2024-02-23T00:23:00.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.960+0000 7f86dbfff700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 0x7f86e00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:00.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.960+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f4045ad0 con 0x7f86f4070430 2024-02-23T00:23:00.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.960+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f86e8041450 con 0x7f86f4070430 2024-02-23T00:23:00.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.961+0000 7f86f2ffd700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 0x7f86e00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:00.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.961+0000 7f86f2ffd700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 0x7f86e00322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f86e40096a0 tx=0x7f86e4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:00.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:00.965+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86e8016070 con 0x7f86f4070430 2024-02-23T00:23:01.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:59.873189+0000 mgr.smithi090.pxxmlz (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:01.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:22:59.873890+0000 mgr.smithi090.pxxmlz (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-23T00:23:01.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:22:59.876147+0000 mon.smithi090 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:01.142 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.140+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f86f410bfe0 con 0x7f86e002fe10 2024-02-23T00:23:01.145 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.144+0000 7f86dbfff700 1 -- 172.21.15.90:0/3944261155 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f86f410bfe0 con 0x7f86e002fe10 2024-02-23T00:23:01.145 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.145+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 msgr2=0x7f86e00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.145+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 0x7f86e00322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f86e40096a0 tx=0x7f86e4006b40).stop 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.145+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 msgr2=0x7f86f410ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.145+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f86e8006800 tx=0x7f86e8006ee0).stop 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.145+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 shutdown_connections 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.146+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f86e002fe10 0x7f86e00322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.146+0000 7f86f9eef700 1 --2- 172.21.15.90:0/3944261155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86f4070430 0x7f86f410ab40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.146+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 >> 172.21.15.90:0/3944261155 conn(0x7f86f40f6530 msgr2=0x7f86f40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.146+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 shutdown_connections 2024-02-23T00:23:01.148 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:01.146+0000 7f86f9eef700 1 -- 172.21.15.90:0/3944261155 wait complete. 2024-02-23T00:23:01.773 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying crash service with default placement... 2024-02-23T00:23:01.773 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-23T00:23:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.259+0000 7f69d2270700 1 Processor -- start 2024-02-23T00:23:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.259+0000 7f69d2270700 1 -- start start 2024-02-23T00:23:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.259+0000 7f69d2270700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.259+0000 7f69d2270700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc070790 con 0x7f69cc070370 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38500/0 (socket says 172.21.15.90:38500) 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69cbfff700 1 -- 172.21.15.90:0/2104381205 learned_addr learned my addr 172.21.15.90:0/2104381205 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69cbfff700 1 -- 172.21.15.90:0/2104381205 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0ff000 con 0x7f69cc070370 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69cbfff700 1 --2- 172.21.15.90:0/2104381205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f69bc00b5f0 tx=0x7f69bc006ee0).ready entity=mon.0 client_cookie=fe0fc19d80a566c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69caffd700 1 -- 172.21.15.90:0/2104381205 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc00d600 con 0x7f69cc070370 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69caffd700 1 -- 172.21.15.90:0/2104381205 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f69bc00dbf0 con 0x7f69cc070370 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.260+0000 7f69caffd700 1 -- 172.21.15.90:0/2104381205 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc011cb0 con 0x7f69cc070370 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 -- 172.21.15.90:0/2104381205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 msgr2=0x7f69cc0feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 --2- 172.21.15.90:0/2104381205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f69bc00b5f0 tx=0x7f69bc006ee0).stop 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 -- 172.21.15.90:0/2104381205 shutdown_connections 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 --2- 172.21.15.90:0/2104381205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc0feb10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 -- 172.21.15.90:0/2104381205 >> 172.21.15.90:0/2104381205 conn(0x7f69cc0f6530 msgr2=0x7f69cc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 -- 172.21.15.90:0/2104381205 shutdown_connections 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.261+0000 7f69d2270700 1 -- 172.21.15.90:0/2104381205 wait complete. 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69d2270700 1 Processor -- start 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69d2270700 1 -- start start 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69d2270700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69d2270700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc10b020 con 0x7f69cc070370 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69cbfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38506/0 (socket says 172.21.15.90:38506) 2024-02-23T00:23:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69cbfff700 1 -- 172.21.15.90:0/4013956279 learned_addr learned my addr 172.21.15.90:0/4013956279 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.262+0000 7f69cbfff700 1 -- 172.21.15.90:0/4013956279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc00b200 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69cbfff700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f69bc006800 tx=0x7f69bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc01b570 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f69bc01bb60 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69bc01a9f0 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc10b220 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc10b6b0 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.263+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f69bc00f070 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.264+0000 7f69c8ff9700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 0x7f69b8032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.264+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f69bc0413a0 con 0x7f69cc070370 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.264+0000 7f69cb7fe700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 0x7f69b8032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.264+0000 7f69cb7fe700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 0x7f69b8032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f69c00096a0 tx=0x7f69c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:02.268 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.264+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc045ad0 con 0x7f69cc070370 2024-02-23T00:23:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.268+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69bc025460 con 0x7f69cc070370 2024-02-23T00:23:02.443 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.442+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f69cc104bf0 con 0x7f69b802fe70 2024-02-23T00:23:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.446+0000 7f69c8ff9700 1 -- 172.21.15.90:0/4013956279 <== mgr.14114 v2:172.21.15.90:6800/3107336444 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f69cc104bf0 con 0x7f69b802fe70 2024-02-23T00:23:02.447 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 msgr2=0x7f69b8032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 0x7f69b8032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f69c00096a0 tx=0x7f69c0006b40).stop 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 msgr2=0x7f69cc10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f69bc006800 tx=0x7f69bc006ee0).stop 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 shutdown_connections 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7f69b802fe70 0x7f69b8032330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 --2- 172.21.15.90:0/4013956279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69cc070370 0x7f69cc10ab30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.447+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 >> 172.21.15.90:0/4013956279 conn(0x7f69cc0f6530 msgr2=0x7f69cc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.448+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 shutdown_connections 2024-02-23T00:23:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:02.448+0000 7f69d2270700 1 -- 172.21.15.90:0/4013956279 wait complete. 2024-02-23T00:23:02.472 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:01.142249+0000 mgr.smithi090.pxxmlz (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:02.472 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:01.142771+0000 mgr.smithi090.pxxmlz (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-23T00:23:02.472 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:01.144848+0000 mon.smithi090 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:03.090 INFO:teuthology.orchestra.run.smithi090.stderr:Enabling mgr prometheus module... 2024-02-23T00:23:03.091 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.624+0000 7fa2304c6700 1 Processor -- start 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.624+0000 7fa2304c6700 1 -- start start 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.624+0000 7fa2304c6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.624+0000 7fa2304c6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280fc950 con 0x7fa228070440 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22e262700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22e262700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39798/0 (socket says 172.21.15.90:39798) 2024-02-23T00:23:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22e262700 1 -- 172.21.15.90:0/636793492 learned_addr learned my addr 172.21.15.90:0/636793492 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22e262700 1 -- 172.21.15.90:0/636793492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228101450 con 0x7fa228070440 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22e262700 1 --2- 172.21.15.90:0/636793492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa22000b5f0 tx=0x7fa220006ee0).ready entity=mon.0 client_cookie=a8183207789b0648 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22d260700 1 -- 172.21.15.90:0/636793492 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22000d600 con 0x7fa228070440 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22d260700 1 -- 172.21.15.90:0/636793492 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa22000dbf0 con 0x7fa228070440 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.625+0000 7fa22d260700 1 -- 172.21.15.90:0/636793492 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa220011cb0 con 0x7fa228070440 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 -- 172.21.15.90:0/636793492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 msgr2=0x7fa228100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 --2- 172.21.15.90:0/636793492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa22000b5f0 tx=0x7fa220006ee0).stop 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 -- 172.21.15.90:0/636793492 shutdown_connections 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 --2- 172.21.15.90:0/636793492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa228100f60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 -- 172.21.15.90:0/636793492 >> 172.21.15.90:0/636793492 conn(0x7fa2280f65c0 msgr2=0x7fa2280f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 -- 172.21.15.90:0/636793492 shutdown_connections 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 -- 172.21.15.90:0/636793492 wait complete. 2024-02-23T00:23:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.626+0000 7fa2304c6700 1 Processor -- start 2024-02-23T00:23:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa2304c6700 1 -- start start 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa2304c6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa2304c6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22810d300 con 0x7fa228070440 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa22e262700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa22e262700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39812/0 (socket says 172.21.15.90:39812) 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa22e262700 1 -- 172.21.15.90:0/393310934 learned_addr learned my addr 172.21.15.90:0/393310934 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa22e262700 1 -- 172.21.15.90:0/393310934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22000b200 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa22e262700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa220006800 tx=0x7fa220006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22001b570 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.627+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa22001bb60 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.628+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22001a940 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.628+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22810d500 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.628+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22810e110 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.628+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa228059100 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.629+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa22000f070 con 0x7fa228070440 2024-02-23T00:23:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.629+0000 7fa226ffd700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 0x7fa2140322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.629+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa220041800 con 0x7fa228070440 2024-02-23T00:23:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.629+0000 7fa22da61700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 0x7fa2140322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.630+0000 7fa22da61700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 0x7fa2140322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa21c0096a0 tx=0x7fa21c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.633+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa220010360 con 0x7fa228070440 2024-02-23T00:23:03.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:02.443766+0000 mgr.smithi090.pxxmlz (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:03.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:02.444456+0000 mgr.smithi090.pxxmlz (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-23T00:23:03.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:02.446794+0000 mon.smithi090 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.90:0/3291255926' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:03.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:03.149128+0000 mgr.smithi090.pxxmlz (mgr.14114) 16 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:03.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:03.817+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fa22810ef40 con 0x7fa228070440 2024-02-23T00:23:04.452 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.450+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fa22001ac00 con 0x7fa228070440 2024-02-23T00:23:04.453 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.451+0000 7fa226ffd700 1 -- 172.21.15.90:0/393310934 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fa22000ea90 con 0x7fa228070440 2024-02-23T00:23:04.455 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 msgr2=0x7fa2140322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 0x7fa2140322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa21c0096a0 tx=0x7fa21c006b40).stop 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 msgr2=0x7fa22810ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa220006800 tx=0x7fa220006ee0).stop 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 shutdown_connections 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7fa21402fe10 0x7fa2140322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 --2- 172.21.15.90:0/393310934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa228070440 0x7fa22810ce10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 >> 172.21.15.90:0/393310934 conn(0x7fa2280f65c0 msgr2=0x7fa2280f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:04.457 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 shutdown_connections 2024-02-23T00:23:04.458 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:04.454+0000 7fa2304c6700 1 -- 172.21.15.90:0/393310934 wait complete. 2024-02-23T00:23:04.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:03.818346+0000 mon.smithi090 (mon.0) 59 : audit [INF] from='client.? 172.21.15.90:0/393310934' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-23T00:23:05.132 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying prometheus service with default placement... 2024-02-23T00:23:05.132 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.693+0000 7efca8b58700 1 Processor -- start 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca8b58700 1 -- start start 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca8b58700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca8b58700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca40fadd0 con 0x7efca40fa4c0 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:05.740 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39836/0 (socket says 172.21.15.90:39836) 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca259c700 1 -- 172.21.15.90:0/3722773761 learned_addr learned my addr 172.21.15.90:0/3722773761 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.694+0000 7efca259c700 1 -- 172.21.15.90:0/3722773761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca40faf10 con 0x7efca40fa4c0 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.695+0000 7efca259c700 1 --2- 172.21.15.90:0/3722773761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7efc9400c700 tx=0x7efc94006ee0).ready entity=mon.0 client_cookie=14dfd93914dd4c8a server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.695+0000 7efca1d9b700 1 -- 172.21.15.90:0/3722773761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc94010780 con 0x7efca40fa4c0 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.695+0000 7efca1d9b700 1 -- 172.21.15.90:0/3722773761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efc94010d70 con 0x7efca40fa4c0 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.695+0000 7efca1d9b700 1 -- 172.21.15.90:0/3722773761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc9400ee10 con 0x7efca40fa4c0 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 -- 172.21.15.90:0/3722773761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 msgr2=0x7efca40fa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 --2- 172.21.15.90:0/3722773761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7efc9400c700 tx=0x7efc94006ee0).stop 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 -- 172.21.15.90:0/3722773761 shutdown_connections 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 --2- 172.21.15.90:0/3722773761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca40fa8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:05.741 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 -- 172.21.15.90:0/3722773761 >> 172.21.15.90:0/3722773761 conn(0x7efca40f5cc0 msgr2=0x7efca40f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 -- 172.21.15.90:0/3722773761 shutdown_connections 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.696+0000 7efca8b58700 1 -- 172.21.15.90:0/3722773761 wait complete. 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca8b58700 1 Processor -- start 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca8b58700 1 -- start start 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca8b58700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca8b58700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca410b1a0 con 0x7efca40fa4c0 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39842/0 (socket says 172.21.15.90:39842) 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.697+0000 7efca259c700 1 -- 172.21.15.90:0/1253201456 learned_addr learned my addr 172.21.15.90:0/1253201456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efca259c700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9400c310 con 0x7efca40fa4c0 2024-02-23T00:23:05.742 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efca259c700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7efc9400e5d0 tx=0x7efc94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc94019630 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efc9401d070 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca410b3a0 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc94018a00 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.698+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca410a400 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.699+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc88004fa0 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.700+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7efc9400e040 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.700+0000 7efc9b7fe700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.700+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efc94041980 con 0x7efca40fa4c0 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.701+0000 7efc99d9b700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3107336444 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.701+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:23:05.743 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.707+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc9400fbb0 con 0x7efca40fa4c0 2024-02-23T00:23:05.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:04.452668+0000 mon.smithi090 (mon.0) 60 : audit [INF] from='client.? 172.21.15.90:0/393310934' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-23T00:23:05.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:04.452723+0000 mon.smithi090 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi090.pxxmlz(active, since 21s) 2024-02-23T00:23:05.903 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.902+0000 7efc99d9b700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3107336444 2024-02-23T00:23:05.904 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.902+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:23:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:05.913+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efc88000bc0 con 0x7efc8402fec0 2024-02-23T00:23:06.304 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:06.303+0000 7efc99d9b700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3107336444 2024-02-23T00:23:06.305 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:06.303+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-23T00:23:07.106 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:07.104+0000 7efc99d9b700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3107336444 2024-02-23T00:23:07.106 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:07.104+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-23T00:23:08.707 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:08.705+0000 7efc99d9b700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3107336444 2024-02-23T00:23:08.707 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:08.705+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-23T00:23:09.484 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:09.482+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7efc94018b60 con 0x7efca40fa4c0 2024-02-23T00:23:09.484 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:09.482+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 msgr2=0x7efc84032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:23:09.485 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:09.482+0000 7efc9b7fe700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:09.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:09.431767+0000 mon.smithi090 (mon.0) 62 : cluster [INF] Active manager daemon smithi090.pxxmlz restarted 2024-02-23T00:23:09.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:09.432141+0000 mon.smithi090 (mon.0) 63 : cluster [INF] Activating manager daemon smithi090.pxxmlz 2024-02-23T00:23:09.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:09.433994+0000 mon.smithi090 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-23T00:23:10.496 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.493+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7efc940436b0 con 0x7efca40fa4c0 2024-02-23T00:23:10.497 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.494+0000 7efc9b7fe700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 0x7efc840360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:10.497 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.494+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 --> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efc88000bc0 con 0x7efc84033ce0 2024-02-23T00:23:10.497 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.494+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 0x7efc840360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:10.497 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.494+0000 7efc99d9b700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 0x7efc840360d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efc8c0036d0 tx=0x7efc8c006d60).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:10.503 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.502+0000 7efc9b7fe700 1 -- 172.21.15.90:0/1253201456 <== mgr.14142 v2:172.21.15.90:6800/614814404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7efc88000bc0 con 0x7efc84033ce0 2024-02-23T00:23:10.503 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-23T00:23:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 msgr2=0x7efc840360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 0x7efc840360d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efc8c0036d0 tx=0x7efc8c006d60).stop 2024-02-23T00:23:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 msgr2=0x7efca4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:10.506 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7efc9400e5d0 tx=0x7efc94006ee0).stop 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 shutdown_connections 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efc84033ce0 0x7efc840360d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efca40fa4c0 0x7efca4109cd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 --2- 172.21.15.90:0/1253201456 >> [v2:172.21.15.90:6800/3107336444,v1:172.21.15.90:6801/3107336444] conn(0x7efc8402fec0 0x7efc84032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.504+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 >> 172.21.15.90:0/1253201456 conn(0x7efca40f5cc0 msgr2=0x7efca40f66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.505+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 shutdown_connections 2024-02-23T00:23:10.507 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:10.505+0000 7efca8b58700 1 -- 172.21.15.90:0/1253201456 wait complete. 2024-02-23T00:23:10.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:09.484545+0000 mon.smithi090 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi090.pxxmlz(active, starting, since 0.0524849s) 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.487316+0000 mon.smithi090 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.487918+0000 mon.smithi090 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pxxmlz", "id": "smithi090.pxxmlz"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.488548+0000 mon.smithi090 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.488743+0000 mon.smithi090 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.488926+0000 mon.smithi090 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:09.502235+0000 mon.smithi090 (mon.0) 71 : cluster [INF] Manager daemon smithi090.pxxmlz is now available 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.598619+0000 mon.smithi090 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.600339+0000 mon.smithi090 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.614753+0000 mon.smithi090 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:23:10.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:09.622331+0000 mon.smithi090 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:23:11.089 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying grafana service with default placement... 2024-02-23T00:23:11.089 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.676+0000 7f8a84f00700 1 Processor -- start 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.676+0000 7f8a84f00700 1 -- start start 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.676+0000 7f8a84f00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.676+0000 7f8a84f00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800728f0 con 0x7f8a800724d0 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39898/0 (socket says 172.21.15.90:39898) 2024-02-23T00:23:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7e59c700 1 -- 172.21.15.90:0/3120724751 learned_addr learned my addr 172.21.15.90:0/3120724751 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7e59c700 1 -- 172.21.15.90:0/3120724751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80070f60 con 0x7f8a800724d0 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7e59c700 1 --2- 172.21.15.90:0/3120724751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a6800b5f0 tx=0x7f8a68006ee0).ready entity=mon.0 client_cookie=b834a6c930941ce3 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7d59a700 1 -- 172.21.15.90:0/3120724751 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6800d600 con 0x7f8a800724d0 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7d59a700 1 -- 172.21.15.90:0/3120724751 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8a6800dbf0 con 0x7f8a800724d0 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.677+0000 7f8a7d59a700 1 -- 172.21.15.90:0/3120724751 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a68011cb0 con 0x7f8a800724d0 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 -- 172.21.15.90:0/3120724751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 msgr2=0x7f8a80070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 --2- 172.21.15.90:0/3120724751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8a6800b5f0 tx=0x7f8a68006ee0).stop 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 -- 172.21.15.90:0/3120724751 shutdown_connections 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 --2- 172.21.15.90:0/3120724751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a80070a70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 -- 172.21.15.90:0/3120724751 >> 172.21.15.90:0/3120724751 conn(0x7f8a800f7640 msgr2=0x7f8a800f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 -- 172.21.15.90:0/3120724751 shutdown_connections 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 -- 172.21.15.90:0/3120724751 wait complete. 2024-02-23T00:23:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.678+0000 7f8a84f00700 1 Processor -- start 2024-02-23T00:23:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a84f00700 1 -- start start 2024-02-23T00:23:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a84f00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a84f00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8010d270 con 0x7f8a800724d0 2024-02-23T00:23:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39910/0 (socket says 172.21.15.90:39910) 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a7e59c700 1 -- 172.21.15.90:0/1443393739 learned_addr learned my addr 172.21.15.90:0/1443393739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a7e59c700 1 -- 172.21.15.90:0/1443393739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6800b200 con 0x7f8a800724d0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a7e59c700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8a68006800 tx=0x7f8a68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6801b570 con 0x7f8a800724d0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8010d470 con 0x7f8a800724d0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8a6801bb60 con 0x7f8a800724d0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6801a9f0 con 0x7f8a800724d0 2024-02-23T00:23:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.679+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8010e0b0 con 0x7f8a800724d0 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.680+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8a6800f070 con 0x7f8a800724d0 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.680+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8010d900 con 0x7f8a800724d0 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.680+0000 7f8a76ffd700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 0x7f8a600323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.680+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8a680415e0 con 0x7f8a800724d0 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.681+0000 7f8a7dd9b700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 0x7f8a600323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:11.685 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.681+0000 7f8a7dd9b700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 0x7f8a600323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:11.686 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.684+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a68022310 con 0x7f8a800724d0 2024-02-23T00:23:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:10.496987+0000 mon.smithi090 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi090.pxxmlz(active, since 1.06492s) 2024-02-23T00:23:11.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:10.502527+0000 mon.smithi090 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:11.865 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.863+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 --> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a8010e3b0 con 0x7f8a6002ff20 2024-02-23T00:23:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.869+0000 7f8a76ffd700 1 -- 172.21.15.90:0/1443393739 <== mgr.14142 v2:172.21.15.90:6800/614814404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8a8010e3b0 con 0x7f8a6002ff20 2024-02-23T00:23:11.871 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-23T00:23:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 msgr2=0x7f8a600323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 0x7f8a600323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2024-02-23T00:23:11.873 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 msgr2=0x7f8a8010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8a68006800 tx=0x7f8a68006ee0).stop 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 shutdown_connections 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8a6002ff20 0x7f8a600323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 --2- 172.21.15.90:0/1443393739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a800724d0 0x7f8a8010cd80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 >> 172.21.15.90:0/1443393739 conn(0x7f8a800f7640 msgr2=0x7f8a800f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 shutdown_connections 2024-02-23T00:23:11.874 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:11.871+0000 7f8a84f00700 1 -- 172.21.15.90:0/1443393739 wait complete. 2024-02-23T00:23:12.403 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying node-exporter service with default placement... 2024-02-23T00:23:12.404 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-23T00:23:12.669 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:10.497667+0000 mgr.smithi090.pxxmlz (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-23T00:23:12.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:11.506911+0000 mon.smithi090 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi090.pxxmlz(active, since 2s) 2024-02-23T00:23:12.670 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:11.870383+0000 mon.smithi090 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.959+0000 7efe8e061700 1 Processor -- start 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.959+0000 7efe8e061700 1 -- start start 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe8e061700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe8e061700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe880ff070 con 0x7efe88070420 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39918/0 (socket says 172.21.15.90:39918) 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe877fe700 1 -- 172.21.15.90:0/71878086 learned_addr learned my addr 172.21.15.90:0/71878086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe877fe700 1 -- 172.21.15.90:0/71878086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe880ff1b0 con 0x7efe88070420 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.960+0000 7efe877fe700 1 --2- 172.21.15.90:0/71878086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efe7c00b5f0 tx=0x7efe7c006ee0).ready entity=mon.0 client_cookie=e925554eff75d71f server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe867fc700 1 -- 172.21.15.90:0/71878086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe7c00d600 con 0x7efe88070420 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe867fc700 1 -- 172.21.15.90:0/71878086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efe7c00dbf0 con 0x7efe88070420 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe867fc700 1 -- 172.21.15.90:0/71878086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe7c011cb0 con 0x7efe88070420 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 -- 172.21.15.90:0/71878086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 msgr2=0x7efe880feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 --2- 172.21.15.90:0/71878086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efe7c00b5f0 tx=0x7efe7c006ee0).stop 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 -- 172.21.15.90:0/71878086 shutdown_connections 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 --2- 172.21.15.90:0/71878086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe880feb80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 -- 172.21.15.90:0/71878086 >> 172.21.15.90:0/71878086 conn(0x7efe880f65c0 msgr2=0x7efe880f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 -- 172.21.15.90:0/71878086 shutdown_connections 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.961+0000 7efe8e061700 1 -- 172.21.15.90:0/71878086 wait complete. 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe8e061700 1 Processor -- start 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe8e061700 1 -- start start 2024-02-23T00:23:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe8e061700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe8e061700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8810b080 con 0x7efe88070420 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe877fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39920/0 (socket says 172.21.15.90:39920) 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.962+0000 7efe877fe700 1 -- 172.21.15.90:0/1663739523 learned_addr learned my addr 172.21.15.90:0/1663739523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe877fe700 1 -- 172.21.15.90:0/1663739523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c00b200 con 0x7efe88070420 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe877fe700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efe7c006800 tx=0x7efe7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe7c01b570 con 0x7efe88070420 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efe7c01bb60 con 0x7efe88070420 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8810b280 con 0x7efe88070420 2024-02-23T00:23:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe7c01a9f0 con 0x7efe88070420 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.963+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8810b710 con 0x7efe88070420 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7efe7c00f070 con 0x7efe88070420 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe6ffff700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 0x7efe74032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7efe7c0423d0 con 0x7efe88070420 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe86ffd700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 0x7efe74032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe88045ad0 con 0x7efe88070420 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.964+0000 7efe86ffd700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 0x7efe74032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efe780096a0 tx=0x7efe78006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:12.968+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe7c016070 con 0x7efe88070420 2024-02-23T00:23:13.145 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.144+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 --> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7efe880008d0 con 0x7efe7402ff70 2024-02-23T00:23:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.147+0000 7efe6ffff700 1 -- 172.21.15.90:0/1663739523 <== mgr.14142 v2:172.21.15.90:6800/614814404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7efe880008d0 con 0x7efe7402ff70 2024-02-23T00:23:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-23T00:23:13.152 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 msgr2=0x7efe74032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 0x7efe74032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7efe780096a0 tx=0x7efe78006b40).stop 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 msgr2=0x7efe8810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efe7c006800 tx=0x7efe7c006ee0).stop 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 shutdown_connections 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7efe7402ff70 0x7efe74032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 --2- 172.21.15.90:0/1663739523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe88070420 0x7efe8810ab90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 >> 172.21.15.90:0/1663739523 conn(0x7efe880f65c0 msgr2=0x7efe880f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 shutdown_connections 2024-02-23T00:23:13.154 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:13.149+0000 7efe8e061700 1 -- 172.21.15.90:0/1663739523 wait complete. 2024-02-23T00:23:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:Deploying alertmanager service with default placement... 2024-02-23T00:23:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-23T00:23:13.764 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:11.864831+0000 mgr.smithi090.pxxmlz (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:13.764 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:11.865375+0000 mgr.smithi090.pxxmlz (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-23T00:23:13.765 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:13.148397+0000 mon.smithi090 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.244+0000 7fb97a93a700 1 Processor -- start 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.244+0000 7fb97a93a700 1 -- start start 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.244+0000 7fb97a93a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.244+0000 7fb97a93a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740fb6a0 con 0x7fb9740fad90 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.244+0000 7fb973fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb973fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32780/0 (socket says 172.21.15.90:32780) 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb973fff700 1 -- 172.21.15.90:0/11778432 learned_addr learned my addr 172.21.15.90:0/11778432 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb973fff700 1 -- 172.21.15.90:0/11778432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740fb7e0 con 0x7fb9740fad90 2024-02-23T00:23:14.249 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb973fff700 1 --2- 172.21.15.90:0/11778432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb95c00b5f0 tx=0x7fb95c006ee0).ready entity=mon.0 client_cookie=c2afbe1fb51a6bcb server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb972ffd700 1 -- 172.21.15.90:0/11778432 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c00d600 con 0x7fb9740fad90 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb972ffd700 1 -- 172.21.15.90:0/11778432 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb95c00dbf0 con 0x7fb9740fad90 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.245+0000 7fb972ffd700 1 -- 172.21.15.90:0/11778432 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c011cb0 con 0x7fb9740fad90 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 -- 172.21.15.90:0/11778432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 msgr2=0x7fb9740fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 --2- 172.21.15.90:0/11778432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb95c00b5f0 tx=0x7fb95c006ee0).stop 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 -- 172.21.15.90:0/11778432 shutdown_connections 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 --2- 172.21.15.90:0/11778432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb9740fb1b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 -- 172.21.15.90:0/11778432 >> 172.21.15.90:0/11778432 conn(0x7fb9740f6530 msgr2=0x7fb9740f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 -- 172.21.15.90:0/11778432 shutdown_connections 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.246+0000 7fb97a93a700 1 -- 172.21.15.90:0/11778432 wait complete. 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb97a93a700 1 Processor -- start 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb97a93a700 1 -- start start 2024-02-23T00:23:14.250 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb97a93a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb97a93a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97410d300 con 0x7fb9740fad90 2024-02-23T00:23:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb973fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb973fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32786/0 (socket says 172.21.15.90:32786) 2024-02-23T00:23:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb973fff700 1 -- 172.21.15.90:0/227342051 learned_addr learned my addr 172.21.15.90:0/227342051 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.247+0000 7fb973fff700 1 -- 172.21.15.90:0/227342051 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c00b200 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb973fff700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fb95c00bf40 tx=0x7fb95c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c01b570 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb95c01bb60 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c01aa20 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97410d500 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.248+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97410e110 con 0x7fb9740fad90 2024-02-23T00:23:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb95c00f070 con 0x7fb9740fad90 2024-02-23T00:23:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb974059100 con 0x7fb9740fad90 2024-02-23T00:23:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb970ff9700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 0x7fb9540323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb95c041620 con 0x7fb9740fad90 2024-02-23T00:23:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb9737fe700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 0x7fb9540323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.249+0000 7fb9737fe700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 0x7fb9540323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb9640096a0 tx=0x7fb964006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:14.254 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.253+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb95c019cd0 con 0x7fb9740fad90 2024-02-23T00:23:14.428 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.427+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 --> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb97410e410 con 0x7fb95402ff20 2024-02-23T00:23:14.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.431+0000 7fb970ff9700 1 -- 172.21.15.90:0/227342051 <== mgr.14142 v2:172.21.15.90:6800/614814404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb97410e410 con 0x7fb95402ff20 2024-02-23T00:23:14.432 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 msgr2=0x7fb9540323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 0x7fb9540323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb9640096a0 tx=0x7fb964006b40).stop 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 msgr2=0x7fb97410ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fb95c00bf40 tx=0x7fb95c006ee0).stop 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 shutdown_connections 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fb95402ff20 0x7fb9540323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 --2- 172.21.15.90:0/227342051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9740fad90 0x7fb97410ce10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:14.435 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.432+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 >> 172.21.15.90:0/227342051 conn(0x7fb9740f6530 msgr2=0x7fb974070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:14.436 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.433+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 shutdown_connections 2024-02-23T00:23:14.436 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:14.433+0000 7fb97a93a700 1 -- 172.21.15.90:0/227342051 wait complete. 2024-02-23T00:23:14.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:13.145617+0000 mgr.smithi090.pxxmlz (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:14.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:13.146271+0000 mgr.smithi090.pxxmlz (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-23T00:23:14.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:14.428905+0000 mgr.smithi090.pxxmlz (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:14.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:14.429458+0000 mgr.smithi090.pxxmlz (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-23T00:23:14.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:14.431777+0000 mon.smithi090 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:15.078 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e8cce700 1 Processor -- start 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e8cce700 1 -- start start 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e8cce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e8cce700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40fc8c0 con 0x7fc5e40703b0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32790/0 (socket says 172.21.15.90:32790) 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e259c700 1 -- 172.21.15.90:0/2223086448 learned_addr learned my addr 172.21.15.90:0/2223086448 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.614+0000 7fc5e259c700 1 -- 172.21.15.90:0/2223086448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e41013c0 con 0x7fc5e40703b0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e259c700 1 --2- 172.21.15.90:0/2223086448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc5d400b5f0 tx=0x7fc5d4006ee0).ready entity=mon.0 client_cookie=c9bfbbafee7ab04f server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e159a700 1 -- 172.21.15.90:0/2223086448 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5d400d600 con 0x7fc5e40703b0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e159a700 1 -- 172.21.15.90:0/2223086448 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc5d400dbf0 con 0x7fc5e40703b0 2024-02-23T00:23:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e159a700 1 -- 172.21.15.90:0/2223086448 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5d4011cb0 con 0x7fc5e40703b0 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 -- 172.21.15.90:0/2223086448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 msgr2=0x7fc5e4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/2223086448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc5d400b5f0 tx=0x7fc5d4006ee0).stop 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 -- 172.21.15.90:0/2223086448 shutdown_connections 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/2223086448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e4100ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 -- 172.21.15.90:0/2223086448 >> 172.21.15.90:0/2223086448 conn(0x7fc5e40f6530 msgr2=0x7fc5e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 -- 172.21.15.90:0/2223086448 shutdown_connections 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.615+0000 7fc5e8cce700 1 -- 172.21.15.90:0/2223086448 wait complete. 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e8cce700 1 Processor -- start 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e8cce700 1 -- start start 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e8cce700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e8cce700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e410d310 con 0x7fc5e40703b0 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32798/0 (socket says 172.21.15.90:32798) 2024-02-23T00:23:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e259c700 1 -- 172.21.15.90:0/3310638136 learned_addr learned my addr 172.21.15.90:0/3310638136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.616+0000 7fc5e259c700 1 -- 172.21.15.90:0/3310638136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d400b200 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5e259c700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc5d400df90 tx=0x7fc5d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5d401b410 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc5d401ba00 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e410d510 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5d401aa30 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.617+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e410e120 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc5d400f070 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5d2ffd700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 0x7fc5c80323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e4059100 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc5d4041330 con 0x7fc5e40703b0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5e1d9b700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 0x7fc5c80323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.618+0000 7fc5e1d9b700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 0x7fc5c80323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc5d80096a0 tx=0x7fc5d8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:15.623 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.622+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5d4016070 con 0x7fc5e40703b0 2024-02-23T00:23:15.772 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.771+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc5e4045ad0 con 0x7fc5e40703b0 2024-02-23T00:23:15.775 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.774+0000 7fc5d2ffd700 1 -- 172.21.15.90:0/3310638136 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc5d4022d20 con 0x7fc5e40703b0 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 msgr2=0x7fc5c80323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 0x7fc5c80323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc5d80096a0 tx=0x7fc5d8006b40).stop 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 msgr2=0x7fc5e410ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc5d400df90 tx=0x7fc5d4006ee0).stop 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 shutdown_connections 2024-02-23T00:23:15.778 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7fc5c802ff20 0x7fc5c80323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:15.779 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 --2- 172.21.15.90:0/3310638136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5e40703b0 0x7fc5e410ce20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:15.779 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.776+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 >> 172.21.15.90:0/3310638136 conn(0x7fc5e40f6530 msgr2=0x7fc5e40f8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:15.779 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.777+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 shutdown_connections 2024-02-23T00:23:15.779 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:15.777+0000 7fc5e8cce700 1 -- 172.21.15.90:0/3310638136 wait complete. 2024-02-23T00:23:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a66e20700 1 Processor -- start 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a66e20700 1 -- start start 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a66e20700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a66e20700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600fef40 con 0x7f3a60070430 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a64bbc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a64bbc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32814/0 (socket says 172.21.15.90:32814) 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.914+0000 7f3a64bbc700 1 -- 172.21.15.90:0/3349851295 learned_addr learned my addr 172.21.15.90:0/3349851295 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a64bbc700 1 -- 172.21.15.90:0/3349851295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600ff080 con 0x7f3a60070430 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a64bbc700 1 --2- 172.21.15.90:0/3349851295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f3a5400b5f0 tx=0x7f3a54006ee0).ready entity=mon.0 client_cookie=b694ee207cea3f5e server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a5f7fe700 1 -- 172.21.15.90:0/3349851295 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a5400d600 con 0x7f3a60070430 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a5f7fe700 1 -- 172.21.15.90:0/3349851295 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3a5400dbf0 con 0x7f3a60070430 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a5f7fe700 1 -- 172.21.15.90:0/3349851295 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a54011cb0 con 0x7f3a60070430 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a66e20700 1 -- 172.21.15.90:0/3349851295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 msgr2=0x7f3a600fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a66e20700 1 --2- 172.21.15.90:0/3349851295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f3a5400b5f0 tx=0x7f3a54006ee0).stop 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a66e20700 1 -- 172.21.15.90:0/3349851295 shutdown_connections 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a66e20700 1 --2- 172.21.15.90:0/3349851295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a600fea50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.915+0000 7f3a66e20700 1 -- 172.21.15.90:0/3349851295 >> 172.21.15.90:0/3349851295 conn(0x7f3a600f6530 msgr2=0x7f3a600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 -- 172.21.15.90:0/3349851295 shutdown_connections 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 -- 172.21.15.90:0/3349851295 wait complete. 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 Processor -- start 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 -- start start 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a66e20700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6010b020 con 0x7f3a60070430 2024-02-23T00:23:16.963 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a64bbc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a64bbc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32816/0 (socket says 172.21.15.90:32816) 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.916+0000 7f3a64bbc700 1 -- 172.21.15.90:0/1042495488 learned_addr learned my addr 172.21.15.90:0/1042495488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a64bbc700 1 -- 172.21.15.90:0/1042495488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5400b200 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a64bbc700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f3a54006800 tx=0x7f3a54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a5401b570 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3a5401bb60 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6010b220 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a5401a9f0 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.917+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6010b6b0 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f3a5400f070 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a5d7fa700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 0x7f3a440323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3a54041650 con 0x7f3a60070430 2024-02-23T00:23:16.964 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a5ffff700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 0x7f3a440323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:16.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a60045ad0 con 0x7f3a60070430 2024-02-23T00:23:16.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.918+0000 7f3a5ffff700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 0x7f3a440323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f3a4c0096a0 tx=0x7f3a4c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:16.965 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:16.922+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a54016030 con 0x7f3a60070430 2024-02-23T00:23:17.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:15.775025+0000 mon.smithi090 (mon.0) 82 : audit [INF] from='client.? 172.21.15.90:0/3310638136' entity='client.admin' 2024-02-23T00:23:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.077+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f3a60104b90 con 0x7f3a60070430 2024-02-23T00:23:17.086 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.085+0000 7f3a5d7fa700 1 -- 172.21.15.90:0/1042495488 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f3a54010360 con 0x7f3a60070430 2024-02-23T00:23:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 msgr2=0x7f3a440323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 0x7f3a440323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f3a4c0096a0 tx=0x7f3a4c006b40).stop 2024-02-23T00:23:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 msgr2=0x7f3a6010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f3a54006800 tx=0x7f3a54006ee0).stop 2024-02-23T00:23:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 shutdown_connections 2024-02-23T00:23:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f3a4402ff20 0x7f3a440323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 --2- 172.21.15.90:0/1042495488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a60070430 0x7f3a6010ab30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 >> 172.21.15.90:0/1042495488 conn(0x7f3a600f6530 msgr2=0x7f3a600f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 shutdown_connections 2024-02-23T00:23:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:17.087+0000 7f3a66e20700 1 -- 172.21.15.90:0/1042495488 wait complete. 2024-02-23T00:23:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:Enabling the dashboard module... 2024-02-23T00:23:18.120 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-23T00:23:18.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:17.086058+0000 mon.smithi090 (mon.0) 83 : audit [INF] from='client.? 172.21.15.90:0/1042495488' entity='client.admin' 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c342b700 1 Processor -- start 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c342b700 1 -- start start 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c342b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c342b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0fee00 con 0x7f38bc0703b0 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c2429700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c2429700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32830/0 (socket says 172.21.15.90:32830) 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.348+0000 7f38c2429700 1 -- 172.21.15.90:0/1370477410 learned_addr learned my addr 172.21.15.90:0/1370477410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:19.351 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.349+0000 7f38c2429700 1 -- 172.21.15.90:0/1370477410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0fef40 con 0x7f38bc0703b0 2024-02-23T00:23:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.349+0000 7f38c2429700 1 --2- 172.21.15.90:0/1370477410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f38b800b5f0 tx=0x7f38b8006ee0).ready entity=mon.0 client_cookie=8fcd64e2ac017005 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.349+0000 7f38c1427700 1 -- 172.21.15.90:0/1370477410 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b800d600 con 0x7f38bc0703b0 2024-02-23T00:23:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.349+0000 7f38c1427700 1 -- 172.21.15.90:0/1370477410 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38b800dc30 con 0x7f38bc0703b0 2024-02-23T00:23:19.352 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.349+0000 7f38c1427700 1 -- 172.21.15.90:0/1370477410 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b8011d30 con 0x7f38bc0703b0 2024-02-23T00:23:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 -- 172.21.15.90:0/1370477410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 msgr2=0x7f38bc0fe910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 --2- 172.21.15.90:0/1370477410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f38b800b5f0 tx=0x7f38b8006ee0).stop 2024-02-23T00:23:19.354 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 -- 172.21.15.90:0/1370477410 shutdown_connections 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 --2- 172.21.15.90:0/1370477410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fe910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 -- 172.21.15.90:0/1370477410 >> 172.21.15.90:0/1370477410 conn(0x7f38bc0f6390 msgr2=0x7f38bc0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 -- 172.21.15.90:0/1370477410 shutdown_connections 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 -- 172.21.15.90:0/1370477410 wait complete. 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.350+0000 7f38c342b700 1 Processor -- start 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c342b700 1 -- start start 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c342b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c342b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0fa810 con 0x7f38bc0703b0 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c2429700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c2429700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32838/0 (socket says 172.21.15.90:32838) 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c2429700 1 -- 172.21.15.90:0/785098860 learned_addr learned my addr 172.21.15.90:0/785098860 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.351+0000 7f38c2429700 1 -- 172.21.15.90:0/785098860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b800b200 con 0x7f38bc0703b0 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38c2429700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38b800e040 tx=0x7f38b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:19.355 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b800d9e0 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38b801b8e0 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc0fa9b0 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc0fae40 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.352+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b801a820 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.353+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f38b800f070 con 0x7f38bc0703b0 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.353+0000 7f38aaffd700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 0x7f38a4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:19.356 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.353+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f38b8042320 con 0x7f38bc0703b0 2024-02-23T00:23:19.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.353+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc059100 con 0x7f38bc0703b0 2024-02-23T00:23:19.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.354+0000 7f38c1c28700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 0x7f38a4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:19.357 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.355+0000 7f38c1c28700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 0x7f38a4032430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f38ac0096a0 tx=0x7f38ac006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:19.359 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.358+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38b8016070 con 0x7f38bc0703b0 2024-02-23T00:23:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:19.555+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f38bc045ad0 con 0x7f38bc0703b0 2024-02-23T00:23:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.104+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f38b80257e0 con 0x7f38bc0703b0 2024-02-23T00:23:20.107 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.106+0000 7f38aaffd700 1 -- 172.21.15.90:0/785098860 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f38b8022d20 con 0x7f38bc0703b0 2024-02-23T00:23:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 msgr2=0x7f38a4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 0x7f38a4032430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f38ac0096a0 tx=0x7f38ac006b40).stop 2024-02-23T00:23:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 msgr2=0x7f38bc0fc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38b800e040 tx=0x7f38b8006ee0).stop 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 shutdown_connections 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f38a402ff70 0x7f38a4032430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 --2- 172.21.15.90:0/785098860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38bc0703b0 0x7f38bc0fc390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 >> 172.21.15.90:0/785098860 conn(0x7f38bc0f6390 msgr2=0x7f38bc0f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 shutdown_connections 2024-02-23T00:23:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:20.108+0000 7f38c342b700 1 -- 172.21.15.90:0/785098860 wait complete. 2024-02-23T00:23:20.130 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:18.796567+0000 mon.smithi090 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:20.130 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:19.104566+0000 mon.smithi090 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.90:0/4228577172' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:20.130 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:19.556601+0000 mon.smithi090 (mon.0) 86 : audit [INF] from='client.? 172.21.15.90:0/785098860' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-23T00:23:20.778 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-23T00:23:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.302+0000 7f8f95571700 1 Processor -- start 2024-02-23T00:23:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.302+0000 7f8f95571700 1 -- start start 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f95571700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f95571700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f900fcde0 con 0x7f8f900704a0 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32854/0 (socket says 172.21.15.90:32854) 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8effd700 1 -- 172.21.15.90:0/2074664086 learned_addr learned my addr 172.21.15.90:0/2074664086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8effd700 1 -- 172.21.15.90:0/2074664086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900fcf20 con 0x7f8f900704a0 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8effd700 1 --2- 172.21.15.90:0/2074664086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8f8400b5f0 tx=0x7f8f84006ee0).ready entity=mon.0 client_cookie=d45450640585d331 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8dffb700 1 -- 172.21.15.90:0/2074664086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8400d600 con 0x7f8f900704a0 2024-02-23T00:23:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8dffb700 1 -- 172.21.15.90:0/2074664086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f8400dc30 con 0x7f8f900704a0 2024-02-23T00:23:21.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.303+0000 7f8f8dffb700 1 -- 172.21.15.90:0/2074664086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f84011d30 con 0x7f8f900704a0 2024-02-23T00:23:21.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 -- 172.21.15.90:0/2074664086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 msgr2=0x7f8f900fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:21.319 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 --2- 172.21.15.90:0/2074664086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8f8400b5f0 tx=0x7f8f84006ee0).stop 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 -- 172.21.15.90:0/2074664086 shutdown_connections 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 --2- 172.21.15.90:0/2074664086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f900fc8f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 -- 172.21.15.90:0/2074664086 >> 172.21.15.90:0/2074664086 conn(0x7f8f900f65c0 msgr2=0x7f8f900f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 -- 172.21.15.90:0/2074664086 shutdown_connections 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.304+0000 7f8f95571700 1 -- 172.21.15.90:0/2074664086 wait complete. 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 Processor -- start 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 -- start start 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f901116b0 con 0x7f8f900704a0 2024-02-23T00:23:21.320 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f8effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f8effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32860/0 (socket says 172.21.15.90:32860) 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f8effd700 1 -- 172.21.15.90:0/3992723737 learned_addr learned my addr 172.21.15.90:0/3992723737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f8effd700 1 -- 172.21.15.90:0/3992723737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8400b200 con 0x7f8f900704a0 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f8effd700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8f84011f20 tx=0x7f8f84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8400d960 con 0x7f8f900704a0 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f901118b0 con 0x7f8f900704a0 2024-02-23T00:23:21.321 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f8400dac0 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8401a850 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.305+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f90111d40 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.306+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8f8400f070 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.306+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f90045ad0 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.306+0000 7f8f777fe700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 0x7f8f7c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.306+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8f840413b0 con 0x7f8f900704a0 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.306+0000 7f8f8e7fc700 1 -- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 msgr2=0x7f8f7c032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.307+0000 7f8f8e7fc700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 0x7f8f7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:23:21.322 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.314+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f84022d50 con 0x7f8f900704a0 2024-02-23T00:23:21.509 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.508+0000 7f8f8e7fc700 1 -- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 msgr2=0x7f8f7c032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:21.510 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.508+0000 7f8f8e7fc700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 0x7f8f7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:23:21.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:20.106976+0000 mon.smithi090 (mon.0) 87 : audit [INF] from='client.? 172.21.15.90:0/785098860' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-23T00:23:21.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:20.107054+0000 mon.smithi090 (mon.0) 88 : cluster [DBG] mgrmap e13: smithi090.pxxmlz(active, since 10s) 2024-02-23T00:23:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.542+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8f90112f10 con 0x7f8f900704a0 2024-02-23T00:23:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.542+0000 7f8f777fe700 1 -- 172.21.15.90:0/3992723737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f8f84016070 con 0x7f8f900704a0 2024-02-23T00:23:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:23:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-23T00:23:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "available": true, 2024-02-23T00:23:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "active_name": "smithi090.pxxmlz", 2024-02-23T00:23:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-23T00:23:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 msgr2=0x7f8f7c032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 0x7f8f7c032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 msgr2=0x7f8f901111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8f84011f20 tx=0x7f8f84006ee0).stop 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 shutdown_connections 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f8f7c02ff70 0x7f8f7c032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 --2- 172.21.15.90:0/3992723737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f900704a0 0x7f8f901111c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.544+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 >> 172.21.15.90:0/3992723737 conn(0x7f8f900f65c0 msgr2=0x7f8f900f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.545+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 shutdown_connections 2024-02-23T00:23:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:21.545+0000 7f8f95571700 1 -- 172.21.15.90:0/3992723737 wait complete. 2024-02-23T00:23:22.153 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for the mgr to restart... 2024-02-23T00:23:22.153 INFO:teuthology.orchestra.run.smithi090.stderr:Waiting for mgr epoch 13... 2024-02-23T00:23:22.153 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-23T00:23:22.155 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:21.543406+0000 mon.smithi090 (mon.0) 89 : audit [DBG] from='client.? 172.21.15.90:0/3992723737' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.754+0000 7f1898c52700 1 Processor -- start 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.754+0000 7f1898c52700 1 -- start start 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.754+0000 7f1898c52700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.754+0000 7f1898c52700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940fc980 con 0x7f18940703b0 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32870/0 (socket says 172.21.15.90:32870) 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189259c700 1 -- 172.21.15.90:0/166614471 learned_addr learned my addr 172.21.15.90:0/166614471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189259c700 1 -- 172.21.15.90:0/166614471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894101480 con 0x7f18940703b0 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189259c700 1 --2- 172.21.15.90:0/166614471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f188400b5f0 tx=0x7f1884006ee0).ready entity=mon.0 client_cookie=8c0307b505d4347b server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189159a700 1 -- 172.21.15.90:0/166614471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188400d600 con 0x7f18940703b0 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189159a700 1 -- 172.21.15.90:0/166614471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f188400dc30 con 0x7f18940703b0 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.755+0000 7f189159a700 1 -- 172.21.15.90:0/166614471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1884011d30 con 0x7f18940703b0 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 -- 172.21.15.90:0/166614471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 msgr2=0x7f1894100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 --2- 172.21.15.90:0/166614471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f188400b5f0 tx=0x7f1884006ee0).stop 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 -- 172.21.15.90:0/166614471 shutdown_connections 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 --2- 172.21.15.90:0/166614471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f1894100f90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 -- 172.21.15.90:0/166614471 >> 172.21.15.90:0/166614471 conn(0x7f18940f65b0 msgr2=0x7f18940f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 -- 172.21.15.90:0/166614471 shutdown_connections 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.756+0000 7f1898c52700 1 -- 172.21.15.90:0/166614471 wait complete. 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f1898c52700 1 Processor -- start 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f1898c52700 1 -- start start 2024-02-23T00:23:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f1898c52700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f1898c52700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189410d320 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f189259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f189259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:32872/0 (socket says 172.21.15.90:32872) 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f189259c700 1 -- 172.21.15.90:0/1140527230 learned_addr learned my addr 172.21.15.90:0/1140527230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.757+0000 7f189259c700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188400b200 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f189259c700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f1884011f20 tx=0x7f1884006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188400d960 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f188400dac0 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188401a850 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189410d520 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189410e130 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.758+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f188400f070 con 0x7f18940703b0 2024-02-23T00:23:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.759+0000 7f1882ffd700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.759+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1878032af0 con 0x7f187802ff70 2024-02-23T00:23:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.759+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1884041350 con 0x7f18940703b0 2024-02-23T00:23:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.759+0000 7f1891d9b700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 msgr2=0x7f1878032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.759+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.960+0000 7f1891d9b700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 msgr2=0x7f1878032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:22.962 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:22.960+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:23:23.362 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:23.361+0000 7f1891d9b700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 msgr2=0x7f1878032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:23.363 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:23.361+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-23T00:23:24.163 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:24.162+0000 7f1891d9b700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 msgr2=0x7f1878032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/614814404 2024-02-23T00:23:24.164 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:24.162+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-23T00:23:25.110 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:25.108+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f18840257e0 con 0x7f18940703b0 2024-02-23T00:23:25.110 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:25.108+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 msgr2=0x7f1878032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:23:25.111 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:25.108+0000 7f1882ffd700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:25.057019+0000 mon.smithi090 (mon.0) 90 : cluster [INF] Active manager daemon smithi090.pxxmlz restarted 2024-02-23T00:23:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:25.057408+0000 mon.smithi090 (mon.0) 91 : cluster [INF] Activating manager daemon smithi090.pxxmlz 2024-02-23T00:23:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:25.059612+0000 mon.smithi090 (mon.0) 92 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:26.114 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.112+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f188401aa70 con 0x7f18940703b0 2024-02-23T00:23:26.114 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.112+0000 7f1882ffd700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 0x7f1878035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:26.114 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.112+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1878032af0 con 0x7f1878033c00 2024-02-23T00:23:26.115 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.112+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 0x7f1878035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:26.115 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.113+0000 7f1891d9b700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 0x7f1878035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f18880036d0 tx=0x7f1888006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:26.115 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.113+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f1878032af0 con 0x7f1878033c00 2024-02-23T00:23:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.120+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f189410d9b0 con 0x7f1878033c00 2024-02-23T00:23:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1882ffd700 1 -- 172.21.15.90:0/1140527230 <== mgr.14162 v2:172.21.15.90:6800/1747781840 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f189410d9b0 con 0x7f1878033c00 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: { 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: "initialized": true 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: } 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 msgr2=0x7f1878035ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 0x7f1878035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f18880036d0 tx=0x7f1888006ca0).stop 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 msgr2=0x7f189410ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f1884011f20 tx=0x7f1884006ee0).stop 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 shutdown_connections 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18940703b0 0x7f189410ce30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1878033c00 0x7f1878035ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 --2- 172.21.15.90:0/1140527230 >> [v2:172.21.15.90:6800/614814404,v1:172.21.15.90:6801/614814404] conn(0x7f187802ff70 0x7f1878032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:23:26.124 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 >> 172.21.15.90:0/1140527230 conn(0x7f18940f65b0 msgr2=0x7f18940f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 shutdown_connections 2024-02-23T00:23:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:26.121+0000 7f1898c52700 1 -- 172.21.15.90:0/1140527230 wait complete. 2024-02-23T00:23:26.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:25.109946+0000 mon.smithi090 (mon.0) 93 : cluster [DBG] mgrmap e14: smithi090.pxxmlz(active, starting, since 0.0526236s) 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.110679+0000 mon.smithi090 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.112693+0000 mon.smithi090 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mgr metadata", "who": "smithi090.pxxmlz", "id": "smithi090.pxxmlz"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.112874+0000 mon.smithi090 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.113005+0000 mon.smithi090 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.113111+0000 mon.smithi090 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:25.126342+0000 mon.smithi090 (mon.0) 99 : cluster [INF] Manager daemon smithi090.pxxmlz is now available 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.222486+0000 mon.smithi090 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.224585+0000 mon.smithi090 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.239705+0000 mon.smithi090 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:23:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:25.247746+0000 mon.smithi090 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:23:27.016 INFO:teuthology.orchestra.run.smithi090.stderr:mgr epoch 13 is available 2024-02-23T00:23:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:Generating a dashboard self-signed certificate... 2024-02-23T00:23:27.017 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-23T00:23:27.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:26.114206+0000 mgr.smithi090.pxxmlz (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-23T00:23:27.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:26.114973+0000 mon.smithi090 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi090.pxxmlz(active, since 1.05765s) 2024-02-23T00:23:27.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:26.122058+0000 mgr.smithi090.pxxmlz (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-23T00:23:27.459 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:26.611565+0000 mon.smithi090 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.658+0000 7fc9c76e9700 1 Processor -- start 2024-02-23T00:23:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c76e9700 1 -- start start 2024-02-23T00:23:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c76e9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c76e9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00fc8a0 con 0x7fc9c00703d0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c5485700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c5485700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50508/0 (socket says 172.21.15.90:50508) 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c5485700 1 -- 172.21.15.90:0/2373474182 learned_addr learned my addr 172.21.15.90:0/2373474182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.659+0000 7fc9c5485700 1 -- 172.21.15.90:0/2373474182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c01013a0 con 0x7fc9c00703d0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c5485700 1 --2- 172.21.15.90:0/2373474182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc9b000b5f0 tx=0x7fc9b0006ee0).ready entity=mon.0 client_cookie=dd7acf82ddb3788a server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9affff700 1 -- 172.21.15.90:0/2373474182 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b000d600 con 0x7fc9c00703d0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9affff700 1 -- 172.21.15.90:0/2373474182 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9b000dc30 con 0x7fc9c00703d0 2024-02-23T00:23:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9affff700 1 -- 172.21.15.90:0/2373474182 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b0011d30 con 0x7fc9c00703d0 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c76e9700 1 -- 172.21.15.90:0/2373474182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 msgr2=0x7fc9c0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/2373474182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc9b000b5f0 tx=0x7fc9b0006ee0).stop 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c76e9700 1 -- 172.21.15.90:0/2373474182 shutdown_connections 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/2373474182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c0100eb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.660+0000 7fc9c76e9700 1 -- 172.21.15.90:0/2373474182 >> 172.21.15.90:0/2373474182 conn(0x7fc9c00f6530 msgr2=0x7fc9c00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 -- 172.21.15.90:0/2373474182 shutdown_connections 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 -- 172.21.15.90:0/2373474182 wait complete. 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 Processor -- start 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 -- start start 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c76e9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c010d330 con 0x7fc9c00703d0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c5485700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c5485700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50512/0 (socket says 172.21.15.90:50512) 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.661+0000 7fc9c5485700 1 -- 172.21.15.90:0/3077834608 learned_addr learned my addr 172.21.15.90:0/3077834608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9c5485700 1 -- 172.21.15.90:0/3077834608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b000b200 con 0x7fc9c00703d0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9c5485700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc9b000c010 tx=0x7fc9b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b000da60 con 0x7fc9c00703d0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c010d530 con 0x7fc9c00703d0 2024-02-23T00:23:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9b001b8e0 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b001a990 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.662+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c010e140 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.663+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0059100 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.664+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc9b000f070 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.664+0000 7fc9adffb700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 0x7fc9a8032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.664+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9b0041cf0 con 0x7fc9c00703d0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.664+0000 7fc9c4c84700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 0x7fc9a8032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.664+0000 7fc9c4c84700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 0x7fc9a8032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc9bc0096a0 tx=0x7fc9bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:27.669 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.668+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9b002d980 con 0x7fc9c00703d0 2024-02-23T00:23:27.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.827+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9c010e440 con 0x7fc9a802ff70 2024-02-23T00:23:27.921 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.920+0000 7fc9adffb700 1 -- 172.21.15.90:0/3077834608 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fc9c010e440 con 0x7fc9a802ff70 2024-02-23T00:23:27.921 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 msgr2=0x7fc9a8032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 0x7fc9a8032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc9bc0096a0 tx=0x7fc9bc006b40).stop 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 msgr2=0x7fc9c010ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc9b000c010 tx=0x7fc9b0006ee0).stop 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 shutdown_connections 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc9a802ff70 0x7fc9a8032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 --2- 172.21.15.90:0/3077834608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9c00703d0 0x7fc9c010ce40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.921+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 >> 172.21.15.90:0/3077834608 conn(0x7fc9c00f6530 msgr2=0x7fc9c00f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.922+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 shutdown_connections 2024-02-23T00:23:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:27.922+0000 7fc9c76e9700 1 -- 172.21.15.90:0/3077834608 wait complete. 2024-02-23T00:23:28.767 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:27.612856+0000 mon.smithi090 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi090.pxxmlz(active, since 2s) 2024-02-23T00:23:28.767 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:27.828797+0000 mgr.smithi090.pxxmlz (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:28.767 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:27.918845+0000 mon.smithi090 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:28.767 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:27.920674+0000 mon.smithi090 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:29.277 INFO:teuthology.orchestra.run.smithi090.stderr:Creating initial admin user... 2024-02-23T00:23:29.277 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6u0xjrk2:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09c1a62700 1 Processor -- start 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09c1a62700 1 -- start start 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09c1a62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09c1a62700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0fc7b0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09bb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09bb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50522/0 (socket says 172.21.15.90:50522) 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.824+0000 7f09bb7fe700 1 -- 172.21.15.90:0/2056613972 learned_addr learned my addr 172.21.15.90:0/2056613972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.825+0000 7f09bb7fe700 1 -- 172.21.15.90:0/2056613972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0fc8f0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.825+0000 7f09bb7fe700 1 --2- 172.21.15.90:0/2056613972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f09b000b5f0 tx=0x7f09b0006ee0).ready entity=mon.0 client_cookie=5c4b662ac314eeb3 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.825+0000 7f09ba7fc700 1 -- 172.21.15.90:0/2056613972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b000d600 con 0x7f09bc0fbea0 2024-02-23T00:23:29.829 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.825+0000 7f09ba7fc700 1 -- 172.21.15.90:0/2056613972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09b000dc30 con 0x7f09bc0fbea0 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.825+0000 7f09ba7fc700 1 -- 172.21.15.90:0/2056613972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b0011d30 con 0x7f09bc0fbea0 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- 172.21.15.90:0/2056613972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 msgr2=0x7f09bc0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2056613972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f09b000b5f0 tx=0x7f09b0006ee0).stop 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- 172.21.15.90:0/2056613972 shutdown_connections 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2056613972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc0fc2c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- 172.21.15.90:0/2056613972 >> 172.21.15.90:0/2056613972 conn(0x7f09bc0f76a0 msgr2=0x7f09bc0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- 172.21.15.90:0/2056613972 shutdown_connections 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- 172.21.15.90:0/2056613972 wait complete. 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 Processor -- start 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.826+0000 7f09c1a62700 1 -- start start 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09c1a62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09c1a62700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc10b100 con 0x7f09bc0fbea0 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09bb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09bb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50526/0 (socket says 172.21.15.90:50526) 2024-02-23T00:23:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09bb7fe700 1 -- 172.21.15.90:0/2980307494 learned_addr learned my addr 172.21.15.90:0/2980307494 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09bb7fe700 1 -- 172.21.15.90:0/2980307494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b000b200 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09bb7fe700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f09b0000c00 tx=0x7f09b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b000d9e0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc10b300 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09b001b8e0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b001a7f0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.827+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc10b790 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.828+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f09b000f070 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.828+0000 7f09a3fff700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 0x7f09a8032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.828+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09b0041630 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.828+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc045ad0 con 0x7f09bc0fbea0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.828+0000 7f09baffd700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 0x7f09a8032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.829+0000 7f09baffd700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 0x7f09a8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f09ac0096a0 tx=0x7f09ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.833+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09b0016070 con 0x7f09bc0fbea0 2024-02-23T00:23:29.999 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:29.997+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f09bc058f20 con 0x7f09a802ff70 2024-02-23T00:23:30.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.214+0000 7f09a3fff700 1 -- 172.21.15.90:0/2980307494 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f09bc058f20 con 0x7f09a802ff70 2024-02-23T00:23:30.216 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$zWlfiKKvQ/8nSb03qehN8ecLwlST/KqzD7M/Zgqm9aSXZeuaTc/wO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708647810, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 msgr2=0x7f09a8032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 0x7f09a8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f09ac0096a0 tx=0x7f09ac006b40).stop 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 msgr2=0x7f09bc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f09b0000c00 tx=0x7f09b0006ee0).stop 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 shutdown_connections 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09a802ff70 0x7f09a8032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 --2- 172.21.15.90:0/2980307494 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09bc0fbea0 0x7f09bc10ac10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 >> 172.21.15.90:0/2980307494 conn(0x7f09bc0f76a0 msgr2=0x7f09bc0f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 shutdown_connections 2024-02-23T00:23:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:30.216+0000 7f09c1a62700 1 -- 172.21.15.90:0/2980307494 wait complete. 2024-02-23T00:23:30.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:29.998923+0000 mgr.smithi090.pxxmlz (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:30.835 INFO:teuthology.orchestra.run.smithi090.stderr:Fetching dashboard port number... 2024-02-23T00:23:30.835 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0uizib5l:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpkbjnfme7:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-23T00:23:31.415 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.410+0000 7fbd61d60700 1 Processor -- start 2024-02-23T00:23:31.415 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.410+0000 7fbd61d60700 1 -- start start 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.410+0000 7fbd61d60700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.410+0000 7fbd61d60700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0728b0 con 0x7fbd5c072490 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.410+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50540/0 (socket says 172.21.15.90:50540) 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5b7fe700 1 -- 172.21.15.90:0/1220042416 learned_addr learned my addr 172.21.15.90:0/1220042416 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5b7fe700 1 -- 172.21.15.90:0/1220042416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c070f20 con 0x7fbd5c072490 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5b7fe700 1 --2- 172.21.15.90:0/1220042416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fbd5000b5f0 tx=0x7fbd50006ee0).ready entity=mon.0 client_cookie=e5507ffa7ac5eb44 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5a7fc700 1 -- 172.21.15.90:0/1220042416 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5000d600 con 0x7fbd5c072490 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5a7fc700 1 -- 172.21.15.90:0/1220042416 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd5000dc30 con 0x7fbd5c072490 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.411+0000 7fbd5a7fc700 1 -- 172.21.15.90:0/1220042416 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd50011d30 con 0x7fbd5c072490 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- 172.21.15.90:0/1220042416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 msgr2=0x7fbd5c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 --2- 172.21.15.90:0/1220042416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fbd5000b5f0 tx=0x7fbd50006ee0).stop 2024-02-23T00:23:31.416 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- 172.21.15.90:0/1220042416 shutdown_connections 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 --2- 172.21.15.90:0/1220042416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c070a30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- 172.21.15.90:0/1220042416 >> 172.21.15.90:0/1220042416 conn(0x7fbd5c0f7640 msgr2=0x7fbd5c0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- 172.21.15.90:0/1220042416 shutdown_connections 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- 172.21.15.90:0/1220042416 wait complete. 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 Processor -- start 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.412+0000 7fbd61d60700 1 -- start start 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd61d60700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd61d60700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c10d340 con 0x7fbd5c072490 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd5b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50544/0 (socket says 172.21.15.90:50544) 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd5b7fe700 1 -- 172.21.15.90:0/2231766378 learned_addr learned my addr 172.21.15.90:0/2231766378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd5b7fe700 1 -- 172.21.15.90:0/2231766378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5000b200 con 0x7fbd5c072490 2024-02-23T00:23:31.417 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd5b7fe700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbd5000e040 tx=0x7fbd50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5000d9e0 con 0x7fbd5c072490 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd5001b8e0 con 0x7fbd5c072490 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c10d540 con 0x7fbd5c072490 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5001a860 con 0x7fbd5c072490 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.413+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c10e150 con 0x7fbd5c072490 2024-02-23T00:23:31.418 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.414+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbd5000f070 con 0x7fbd5c072490 2024-02-23T00:23:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.414+0000 7fbd43fff700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 0x7fbd48032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.414+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd50041540 con 0x7fbd5c072490 2024-02-23T00:23:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.414+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c059100 con 0x7fbd5c072490 2024-02-23T00:23:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.414+0000 7fbd5affd700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 0x7fbd48032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.415+0000 7fbd5affd700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 0x7fbd48032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbd4c0096a0 tx=0x7fbd4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.419+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd50016070 con 0x7fbd5c072490 2024-02-23T00:23:31.422 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.421+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd500257e0 con 0x7fbd5c072490 2024-02-23T00:23:31.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:30.215172+0000 mon.smithi090 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:31.571 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.569+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fbd5c045ad0 con 0x7fbd5c072490 2024-02-23T00:23:31.572 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.570+0000 7fbd43fff700 1 -- 172.21.15.90:0/2231766378 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fbd50022d50 con 0x7fbd5c072490 2024-02-23T00:23:31.572 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 8443 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.571+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 msgr2=0x7fbd48032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.571+0000 7fbd61d60700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 0x7fbd48032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbd4c0096a0 tx=0x7fbd4c006b40).stop 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.571+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 msgr2=0x7fbd5c10ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.571+0000 7fbd61d60700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbd5000e040 tx=0x7fbd50006ee0).stop 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 shutdown_connections 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbd4802ff70 0x7fbd48032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 --2- 172.21.15.90:0/2231766378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd5c072490 0x7fbd5c10ce50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 >> 172.21.15.90:0/2231766378 conn(0x7fbd5c0f7640 msgr2=0x7fbd5c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 shutdown_connections 2024-02-23T00:23:31.574 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/ceph: 2024-02-23T00:23:31.572+0000 7fbd61d60700 1 -- 172.21.15.90:0/2231766378 wait complete. 2024-02-23T00:23:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-23T00:23:32.236 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: disabled 2024-02-23T00:23:32.237 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: systemctl is-active firewalld.service 2024-02-23T00:23:32.244 INFO:teuthology.orchestra.run.smithi090.stderr:systemctl: inactive 2024-02-23T00:23:32.244 INFO:teuthology.orchestra.run.smithi090.stderr:firewalld.service is not enabled 2024-02-23T00:23:32.244 INFO:teuthology.orchestra.run.smithi090.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-23T00:23:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:Ceph Dashboard is now available at: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: URL: https://smithi090.front.sepia.ceph.com:8443/ 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: User: admin 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: Password: uef0twi2vg 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:You can access the Ceph CLI with: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.248 INFO:teuthology.orchestra.run.smithi090.stderr: ceph telemetry on 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:For more information see: 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:Bootstrap complete. 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:Releasing lock 140060249237992 on /run/cephadm/898d23fe-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:23:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:Lock 140060249237992 released on /run/cephadm/898d23fe-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:23:32.280 INFO:tasks.cephadm:Fetching config... 2024-02-23T00:23:32.280 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:23:32.281 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-23T00:23:32.339 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-23T00:23:32.339 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:23:32.339 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-23T00:23:32.397 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-23T00:23:32.397 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:23:32.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/mon.smithi090/keyring of=/dev/stdout 2024-02-23T00:23:32.466 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-23T00:23:32.467 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-02-23T00:23:32.467 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-23T00:23:32.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:31.423423+0000 mon.smithi090 (mon.0) 110 : cluster [DBG] mgrmap e17: smithi090.pxxmlz(active, since 6s) 2024-02-23T00:23:32.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:31.571165+0000 mon.smithi090 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.90:0/2231766378' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-23T00:23:32.525 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-23T00:23:32.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfoeZLitAzd3QiNOepmW0noMkEd0yXgN49QijoA4pdW89K55GvZWsT4fIM4cOSBD5papI5vrly5TsHlvwZ9qQsP/CKwbN1ayLehyVtU13zFocyUJqRwZ76Q27Prko20rgkVzWV30OrRUc5KJOGo0AXnUvK5DcHjKEAB1fx4m//Hmxo8qkDyOxqdNHq3RjLcn49AYFadKPcF1JFGX69KVX/a1KB+fwISwlMfdaJAskshhIqKAE+AyC/2YJlPbc0qd2GgCv4VYfm03gNUcgRKUUaT7IXWCDxkMhiG2eJAH9J2fv6PAkvPrt6oaqH4GA8TCLoqhFVrMnuKx8xrP9tNMbZ4GhXLNkoWTLDj69kKwlnQQDCJW/K5VgMKMklbNd4fhqXhqxx1M4WxVMkE2o8rgfA9IOr4XbkRSEK8FeSwScfza3qTi4pFhW/jbVvzn/XlRifaTg3YjNoVfy3TbUwGXSqupF7Fh3hXOG2cw00qVOxKG8/6usz410A0acHaHbZGeE= ceph-898d23fe-d1e1-11ee-95c0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-23T00:23:32.605 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfoeZLitAzd3QiNOepmW0noMkEd0yXgN49QijoA4pdW89K55GvZWsT4fIM4cOSBD5papI5vrly5TsHlvwZ9qQsP/CKwbN1ayLehyVtU13zFocyUJqRwZ76Q27Prko20rgkVzWV30OrRUc5KJOGo0AXnUvK5DcHjKEAB1fx4m//Hmxo8qkDyOxqdNHq3RjLcn49AYFadKPcF1JFGX69KVX/a1KB+fwISwlMfdaJAskshhIqKAE+AyC/2YJlPbc0qd2GgCv4VYfm03gNUcgRKUUaT7IXWCDxkMhiG2eJAH9J2fv6PAkvPrt6oaqH4GA8TCLoqhFVrMnuKx8xrP9tNMbZ4GhXLNkoWTLDj69kKwlnQQDCJW/K5VgMKMklbNd4fhqXhqxx1M4WxVMkE2o8rgfA9IOr4XbkRSEK8FeSwScfza3qTi4pFhW/jbVvzn/XlRifaTg3YjNoVfy3TbUwGXSqupF7Fh3hXOG2cw00qVOxKG8/6usz410A0acHaHbZGeE= ceph-898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:32.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfoeZLitAzd3QiNOepmW0noMkEd0yXgN49QijoA4pdW89K55GvZWsT4fIM4cOSBD5papI5vrly5TsHlvwZ9qQsP/CKwbN1ayLehyVtU13zFocyUJqRwZ76Q27Prko20rgkVzWV30OrRUc5KJOGo0AXnUvK5DcHjKEAB1fx4m//Hmxo8qkDyOxqdNHq3RjLcn49AYFadKPcF1JFGX69KVX/a1KB+fwISwlMfdaJAskshhIqKAE+AyC/2YJlPbc0qd2GgCv4VYfm03gNUcgRKUUaT7IXWCDxkMhiG2eJAH9J2fv6PAkvPrt6oaqH4GA8TCLoqhFVrMnuKx8xrP9tNMbZ4GhXLNkoWTLDj69kKwlnQQDCJW/K5VgMKMklbNd4fhqXhqxx1M4WxVMkE2o8rgfA9IOr4XbkRSEK8FeSwScfza3qTi4pFhW/jbVvzn/XlRifaTg3YjNoVfy3TbUwGXSqupF7Fh3hXOG2cw00qVOxKG8/6usz410A0acHaHbZGeE= ceph-898d23fe-d1e1-11ee-95c0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-23T00:23:32.661 INFO:teuthology.orchestra.run.smithi118.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfoeZLitAzd3QiNOepmW0noMkEd0yXgN49QijoA4pdW89K55GvZWsT4fIM4cOSBD5papI5vrly5TsHlvwZ9qQsP/CKwbN1ayLehyVtU13zFocyUJqRwZ76Q27Prko20rgkVzWV30OrRUc5KJOGo0AXnUvK5DcHjKEAB1fx4m//Hmxo8qkDyOxqdNHq3RjLcn49AYFadKPcF1JFGX69KVX/a1KB+fwISwlMfdaJAskshhIqKAE+AyC/2YJlPbc0qd2GgCv4VYfm03gNUcgRKUUaT7IXWCDxkMhiG2eJAH9J2fv6PAkvPrt6oaqH4GA8TCLoqhFVrMnuKx8xrP9tNMbZ4GhXLNkoWTLDj69kKwlnQQDCJW/K5VgMKMklbNd4fhqXhqxx1M4WxVMkE2o8rgfA9IOr4XbkRSEK8FeSwScfza3qTi4pFhW/jbVvzn/XlRifaTg3YjNoVfy3TbUwGXSqupF7Fh3hXOG2cw00qVOxKG8/6usz410A0acHaHbZGeE= ceph-898d23fe-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:32.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-23T00:23:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 -- 172.21.15.90:0/3581581273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 msgr2=0x7f66840f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 --2- 172.21.15.90:0/3581581273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66840f9150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f6674009fb0 tx=0x7f6674009b30).stop 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 -- 172.21.15.90:0/3581581273 shutdown_connections 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 --2- 172.21.15.90:0/3581581273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66840f9150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 -- 172.21.15.90:0/3581581273 >> 172.21.15.90:0/3581581273 conn(0x7f66840f0b70 msgr2=0x7f66840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 -- 172.21.15.90:0/3581581273 shutdown_connections 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 -- 172.21.15.90:0/3581581273 wait complete. 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.289+0000 7f668a008700 1 Processor -- start 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f668a008700 1 -- start start 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f668a008700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f668a008700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684106c90 con 0x7f6684060220 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f66837fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f66837fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35326/0 (socket says 172.21.15.90:35326) 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f66837fe700 1 -- 172.21.15.90:0/3191260122 learned_addr learned my addr 172.21.15.90:0/3191260122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:23:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f66837fe700 1 -- 172.21.15.90:0/3191260122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6674005040 con 0x7f6684060220 2024-02-23T00:23:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f66837fe700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f66740089f0 tx=0x7f667400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6674009320 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6684106e90 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.290+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684107320 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.291+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6674013380 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.291+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6674014d00 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6664004fa0 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66740263a0 con 0x7f6684060220 2024-02-23T00:23:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f666bfff700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 0x7f666c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6674040e00 con 0x7f6684060220 2024-02-23T00:23:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f6682ffd700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 0x7f666c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.292+0000 7f6682ffd700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 0x7f666c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f66780096a0 tx=0x7f6678006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.296+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66740126f0 con 0x7f6684060220 2024-02-23T00:23:35.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.455+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6664005bf0 con 0x7f6684060220 2024-02-23T00:23:35.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.458+0000 7f666bfff700 1 -- 172.21.15.90:0/3191260122 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6674004490 con 0x7f6684060220 2024-02-23T00:23:35.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 msgr2=0x7f666c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:35.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 0x7f666c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f66780096a0 tx=0x7f6678006b40).stop 2024-02-23T00:23:35.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 msgr2=0x7f66841067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:35.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f66740089f0 tx=0x7f667400b040).stop 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 shutdown_connections 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f666c02ff60 0x7f666c032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 --2- 172.21.15.90:0/3191260122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6684060220 0x7f66841067a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 >> 172.21.15.90:0/3191260122 conn(0x7f66840f0b70 msgr2=0x7f66840f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 shutdown_connections 2024-02-23T00:23:35.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:23:35.460+0000 7f668a008700 1 -- 172.21.15.90:0/3191260122 wait complete. 2024-02-23T00:23:36.066 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi118 2024-02-23T00:23:36.066 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:23:36.066 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.conf 2024-02-23T00:23:36.088 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-02-23T00:23:36.088 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:23:36.149 INFO:tasks.cephadm:Adding host smithi118 to orchestrator... 2024-02-23T00:23:36.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph orch host add smithi118 2024-02-23T00:23:36.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:35.102869+0000 mon.smithi090 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:36.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:35.378302+0000 mon.smithi090 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:23:36.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:35.459414+0000 mon.smithi090 (mon.0) 114 : audit [INF] from='client.? 172.21.15.90:0/3191260122' entity='client.admin' 2024-02-23T00:23:37.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:23:35.384531+0000 mgr.smithi090.pxxmlz (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi090 on smithi090 2024-02-23T00:23:45.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:45.113462+0000 mgr.smithi090.pxxmlz (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:47.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:47.113729+0000 mgr.smithi090.pxxmlz (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:49.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:49.114036+0000 mgr.smithi090.pxxmlz (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:51.114333+0000 mgr.smithi090.pxxmlz (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:53.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:53.114662+0000 mgr.smithi090.pxxmlz (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:55.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:55.114963+0000 mgr.smithi090.pxxmlz (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:57.115319+0000 mgr.smithi090.pxxmlz (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- 172.21.15.118:0/4122230710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 msgr2=0x7f43140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 --2- 172.21.15.118:0/4122230710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f43140f66c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f42fc009fb0 tx=0x7f42fc009b30).stop 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- 172.21.15.118:0/4122230710 shutdown_connections 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 --2- 172.21.15.118:0/4122230710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f43140f66c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- 172.21.15.118:0/4122230710 >> 172.21.15.118:0/4122230710 conn(0x7f43140f1aa0 msgr2=0x7f43140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- 172.21.15.118:0/4122230710 shutdown_connections 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- 172.21.15.118:0/4122230710 wait complete. 2024-02-23T00:23:59.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 Processor -- start 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- start start 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.377+0000 7f4313fff700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4314069e20 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:44402/0 (socket says 172.21.15.118:44402) 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4312ffd700 1 -- 172.21.15.118:0/3274901006 learned_addr learned my addr 172.21.15.118:0/3274901006 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4312ffd700 1 -- 172.21.15.118:0/3274901006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc005040 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4312ffd700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f42fc0089f0 tx=0x7f42fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42fc009320 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4314069fc0 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42fc013380 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42fc014e20 con 0x7f43140f62a0 2024-02-23T00:23:59.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.378+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431406a450 con 0x7f43140f62a0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.379+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42fc0263a0 con 0x7f43140f62a0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.379+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4314059100 con 0x7f43140f62a0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.379+0000 7f430b7fe700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 0x7f4300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.379+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42fc040fa0 con 0x7f43140f62a0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.379+0000 7f43127fc700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 0x7f4300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.380+0000 7f43127fc700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 0x7f4300032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f43040096a0 tx=0x7f4304006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:59.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.384+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42fc01db70 con 0x7f43140f62a0 2024-02-23T00:23:59.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:23:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:23:59.115572+0000 mgr.smithi090.pxxmlz (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:59.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:23:59.555+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}) v1 -- 0x7f431406aca0 con 0x7f430002ff60 2024-02-23T00:24:00.154 INFO:teuthology.orchestra.run.smithi118.stdout:Added host 'smithi118' 2024-02-23T00:24:00.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.152+0000 7f430b7fe700 1 -- 172.21.15.118:0/3274901006 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f431406aca0 con 0x7f430002ff60 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 msgr2=0x7f4300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 0x7f4300032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f43040096a0 tx=0x7f4304006b40).stop 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 msgr2=0x7f431406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f42fc0089f0 tx=0x7f42fc00b040).stop 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 shutdown_connections 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f430002ff60 0x7f4300032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 --2- 172.21.15.118:0/3274901006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43140f62a0 0x7f431406b9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 >> 172.21.15.118:0/3274901006 conn(0x7f43140f1aa0 msgr2=0x7f43140f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:00.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 shutdown_connections 2024-02-23T00:24:00.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:00.154+0000 7f4313fff700 1 -- 172.21.15.118:0/3274901006 wait complete. 2024-02-23T00:24:00.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:23:59.556880+0000 mgr.smithi090.pxxmlz (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:00.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:00.152558+0000 mon.smithi090 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:24:00.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph orch host ls --format=json 2024-02-23T00:24:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:24:00.152913+0000 mgr.smithi090.pxxmlz (mgr.14162) 15 : cephadm [INF] Added host smithi118 2024-02-23T00:24:01.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:01.115844+0000 mgr.smithi090.pxxmlz (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 -- 172.21.15.118:0/230766413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 msgr2=0x7fdac40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 --2- 172.21.15.118:0/230766413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac40f9150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdaac009fb0 tx=0x7fdaac009b30).stop 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 -- 172.21.15.118:0/230766413 shutdown_connections 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 --2- 172.21.15.118:0/230766413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac40f9150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 -- 172.21.15.118:0/230766413 >> 172.21.15.118:0/230766413 conn(0x7fdac40f0b70 msgr2=0x7fdac40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 -- 172.21.15.118:0/230766413 shutdown_connections 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.831+0000 7fdac88a2700 1 -- 172.21.15.118:0/230766413 wait complete. 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac88a2700 1 Processor -- start 2024-02-23T00:24:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac88a2700 1 -- start start 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac88a2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac88a2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac4106c90 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:44420/0 (socket says 172.21.15.118:44420) 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.832+0000 7fdac259c700 1 -- 172.21.15.118:0/2294866516 learned_addr learned my addr 172.21.15.118:0/2294866516 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdac259c700 1 -- 172.21.15.118:0/2294866516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaac005040 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdac259c700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdaac0089f0 tx=0x7fdaac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaac009320 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaac013380 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac4106e90 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaac014e20 con 0x7fdac4060220 2024-02-23T00:24:02.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.833+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac4107320 con 0x7fdac4060220 2024-02-23T00:24:02.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.834+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaac014420 con 0x7fdac4060220 2024-02-23T00:24:02.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.834+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac4045ad0 con 0x7fdac4060220 2024-02-23T00:24:02.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.834+0000 7fdabaffd700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 0x7fdab00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:02.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.834+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaac040f50 con 0x7fdac4060220 2024-02-23T00:24:02.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.834+0000 7fdac1d9b700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 0x7fdab00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:02.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.835+0000 7fdac1d9b700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 0x7fdab00323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdab40096a0 tx=0x7fdab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:02.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:02.839+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaac06d2b0 con 0x7fdac4060220 2024-02-23T00:24:03.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.008+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdac40f1d00 con 0x7fdab002ff10 2024-02-23T00:24:03.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.010+0000 7fdabaffd700 1 -- 172.21.15.118:0/2294866516 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fdac40f1d00 con 0x7fdab002ff10 2024-02-23T00:24:03.013 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:03.013 INFO:teuthology.orchestra.run.smithi118.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "smithi118", "hostname": "smithi118", "labels": [], "status": ""}] 2024-02-23T00:24:03.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 msgr2=0x7fdab00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:03.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 0x7fdab00323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdab40096a0 tx=0x7fdab4006b40).stop 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 msgr2=0x7fdac41067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdaac0089f0 tx=0x7fdaac00b040).stop 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 shutdown_connections 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdab002ff10 0x7fdab00323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 --2- 172.21.15.118:0/2294866516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdac4060220 0x7fdac41067a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 >> 172.21.15.118:0/2294866516 conn(0x7fdac40f0b70 msgr2=0x7fdac40f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 shutdown_connections 2024-02-23T00:24:03.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:03.012+0000 7fdac88a2700 1 -- 172.21.15.118:0/2294866516 wait complete. 2024-02-23T00:24:03.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:03.011617+0000 mgr.smithi090.pxxmlz (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:24:03.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:03.116226+0000 mgr.smithi090.pxxmlz (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:03.700 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-23T00:24:03.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph osd crush tunables default 2024-02-23T00:24:05.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:05.116574+0000 mgr.smithi090.pxxmlz (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 -- 172.21.15.90:0/2247484592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 msgr2=0x7ff8e00f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2247484592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e00f68a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff8d0009fb0 tx=0x7ff8d0009b30).stop 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 -- 172.21.15.90:0/2247484592 shutdown_connections 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2247484592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e00f68a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 -- 172.21.15.90:0/2247484592 >> 172.21.15.90:0/2247484592 conn(0x7ff8e00f1c00 msgr2=0x7ff8e00f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 -- 172.21.15.90:0/2247484592 shutdown_connections 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.479+0000 7ff8e805d700 1 -- 172.21.15.90:0/2247484592 wait complete. 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 Processor -- start 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 -- start start 2024-02-23T00:24:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e0069e20 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e5df9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e5df9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54072/0 (socket says 172.21.15.90:54072) 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e5df9700 1 -- 172.21.15.90:0/2050576846 learned_addr learned my addr 172.21.15.90:0/2050576846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e5df9700 1 -- 172.21.15.90:0/2050576846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0005040 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e5df9700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff8d0005010 tx=0x7ff8d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d0009320 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e0069fc0 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8d0013380 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d0014e50 con 0x7ff8e00f6480 2024-02-23T00:24:05.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.480+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e006a450 con 0x7ff8e00f6480 2024-02-23T00:24:05.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.481+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8d00095f0 con 0x7ff8e00f6480 2024-02-23T00:24:05.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.481+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e0059100 con 0x7ff8e00f6480 2024-02-23T00:24:05.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.482+0000 7ff8d67fc700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 0x7ff8cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:05.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.482+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8d0040e70 con 0x7ff8e00f6480 2024-02-23T00:24:05.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.482+0000 7ff8e55f8700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 0x7ff8cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:05.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.482+0000 7ff8e55f8700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 0x7ff8cc0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff8dc0096a0 tx=0x7ff8dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:05.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.486+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8d0014420 con 0x7ff8e00f6480 2024-02-23T00:24:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:05.642+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff8e00fc090 con 0x7ff8e00f6480 2024-02-23T00:24:06.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.428+0000 7ff8d67fc700 1 -- 172.21.15.90:0/2050576846 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff8d0010230 con 0x7ff8e00f6480 2024-02-23T00:24:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 msgr2=0x7ff8cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 0x7ff8cc0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff8dc0096a0 tx=0x7ff8dc006b40).stop 2024-02-23T00:24:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 msgr2=0x7ff8e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:06.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff8d0005010 tx=0x7ff8d000b040).stop 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 shutdown_connections 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff8cc02ff10 0x7ff8cc0323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 --2- 172.21.15.90:0/2050576846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8e00f6480 0x7ff8e006b9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 >> 172.21.15.90:0/2050576846 conn(0x7ff8e00f1c00 msgr2=0x7ff8e00f2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 shutdown_connections 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-02-23T00:24:06.430+0000 7ff8e805d700 1 -- 172.21.15.90:0/2050576846 wait complete. 2024-02-23T00:24:06.432 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-02-23T00:24:06.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:05.643209+0000 mon.smithi090 (mon.0) 116 : audit [INF] from='client.? 172.21.15.90:0/2050576846' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-23T00:24:07.146 INFO:tasks.cephadm:Adding mon.smithi090 on smithi090 2024-02-23T00:24:07.147 INFO:tasks.cephadm:Adding mon.smithi118 on smithi118 2024-02-23T00:24:07.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply mon '2;smithi090:172.21.15.90=smithi090;smithi118:172.21.15.118=smithi118' 2024-02-23T00:24:07.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:06.429124+0000 mon.smithi090 (mon.0) 117 : audit [INF] from='client.? 172.21.15.90:0/2050576846' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-23T00:24:07.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:06.429171+0000 mon.smithi090 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-23T00:24:07.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:07.116995+0000 mgr.smithi090.pxxmlz (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:09.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/1475797227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 msgr2=0x7fd3f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:09.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/1475797227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f00f90f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fd3e4009fb0 tx=0x7fd3e4009b30).stop 2024-02-23T00:24:09.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/1475797227 shutdown_connections 2024-02-23T00:24:09.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/1475797227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f00f90f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:09.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/1475797227 >> 172.21.15.118:0/1475797227 conn(0x7fd3f00f0b70 msgr2=0x7fd3f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:09.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/1475797227 shutdown_connections 2024-02-23T00:24:09.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/1475797227 wait complete. 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.401+0000 7fd3f6bd9700 1 Processor -- start 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f6bd9700 1 -- start start 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f6bd9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f6bd9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f0106c90 con 0x7fd3f0060280 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f4975700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f4975700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58566/0 (socket says 172.21.15.118:58566) 2024-02-23T00:24:09.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f4975700 1 -- 172.21.15.118:0/4107350964 learned_addr learned my addr 172.21.15.118:0/4107350964 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.402+0000 7fd3f4975700 1 -- 172.21.15.118:0/4107350964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e4005040 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3f4975700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd3e40089f0 tx=0x7fd3e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e4009320 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3e4013380 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f0106e90 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e4014e20 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.403+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f0107320 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3e40095f0 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f0045ad0 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3ed7fa700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3e4040ef0 con 0x7fd3f0060280 2024-02-23T00:24:09.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3effff700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:09.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.404+0000 7fd3effff700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 0x7fd3e00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd3dc0096a0 tx=0x7fd3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:09.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.409+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3e4014420 con 0x7fd3f0060280 2024-02-23T00:24:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:09.117345+0000 mgr.smithi090.pxxmlz (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:09.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.582+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 --> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi090:172.21.15.90=smithi090;smithi118:172.21.15.118=smithi118", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3f00f1ca0 con 0x7fd3e002ff10 2024-02-23T00:24:09.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3ed7fa700 1 -- 172.21.15.118:0/4107350964 <== mgr.14162 v2:172.21.15.90:6800/1747781840 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd3f00f1ca0 con 0x7fd3e002ff10 2024-02-23T00:24:09.590 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mon update... 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 msgr2=0x7fd3e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 0x7fd3e00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd3dc0096a0 tx=0x7fd3dc006b40).stop 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 msgr2=0x7fd3f01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd3e40089f0 tx=0x7fd3e400b040).stop 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 shutdown_connections 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 --2- 172.21.15.118:0/4107350964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3f0060280 0x7fd3f01067a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:09.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 >> 172.21.15.118:0/4107350964 conn(0x7fd3f00f0b70 msgr2=0x7fd3f00f1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:09.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 shutdown_connections 2024-02-23T00:24:09.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:09.589+0000 7fd3f6bd9700 1 -- 172.21.15.118:0/4107350964 wait complete. 2024-02-23T00:24:10.343 DEBUG:teuthology.orchestra.run.smithi118:mon.smithi118> sudo journalctl -f -n 0 -u ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118.service 2024-02-23T00:24:10.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:10.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:10.377 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:-- Logs begin at Fri 2024-02-23 00:03:15 UTC. -- 2024-02-23T00:24:11.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:09.585376+0000 mgr.smithi090.pxxmlz (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi090:172.21.15.90=smithi090;smithi118:172.21.15.118=smithi118", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:11.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:24:09.586730+0000 mgr.smithi090.pxxmlz (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi090:172.21.15.90=smithi090;smithi118:172.21.15.118=smithi118;count:2 2024-02-23T00:24:11.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:09.589396+0000 mon.smithi090 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:24:12.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:11.117592+0000 mgr.smithi090.pxxmlz (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:12.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 -- 172.21.15.118:0/37991491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 msgr2=0x7fa8500f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:12.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 --2- 172.21.15.118:0/37991491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa8500f90f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa840009fb0 tx=0x7fa840009b30).stop 2024-02-23T00:24:12.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 -- 172.21.15.118:0/37991491 shutdown_connections 2024-02-23T00:24:12.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 --2- 172.21.15.118:0/37991491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa8500f90f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 -- 172.21.15.118:0/37991491 >> 172.21.15.118:0/37991491 conn(0x7fa8500f0b70 msgr2=0x7fa8500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 -- 172.21.15.118:0/37991491 shutdown_connections 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.300+0000 7fa857dd0700 1 -- 172.21.15.118:0/37991491 wait complete. 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa857dd0700 1 Processor -- start 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa857dd0700 1 -- start start 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa857dd0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa857dd0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa850106c70 con 0x7fa850060280 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa855b6c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa855b6c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58584/0 (socket says 172.21.15.118:58584) 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa855b6c700 1 -- 172.21.15.118:0/4240969930 learned_addr learned my addr 172.21.15.118:0/4240969930 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.301+0000 7fa855b6c700 1 -- 172.21.15.118:0/4240969930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa840005040 con 0x7fa850060280 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa855b6c700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa8400089f0 tx=0x7fa84000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:12.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa840009320 con 0x7fa850060280 2024-02-23T00:24:12.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa850106e70 con 0x7fa850060280 2024-02-23T00:24:12.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa840013380 con 0x7fa850060280 2024-02-23T00:24:12.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa840014e20 con 0x7fa850060280 2024-02-23T00:24:12.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.302+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa850107300 con 0x7fa850060280 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8400095f0 con 0x7fa850060280 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850045ad0 con 0x7fa850060280 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa8467fc700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 0x7fa83c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa840040f10 con 0x7fa850060280 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa85536b700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 0x7fa83c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:12.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.303+0000 7fa85536b700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 0x7fa83c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:12.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.307+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa840014420 con 0x7fa850060280 2024-02-23T00:24:12.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.511+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8501081b0 con 0x7fa850060280 2024-02-23T00:24:12.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.512+0000 7fa8467fc700 1 -- 172.21.15.118:0/4240969930 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa84001d980 con 0x7fa850060280 2024-02-23T00:24:12.513 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:12.513 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:12.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 msgr2=0x7fa83c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 0x7fa83c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).stop 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 msgr2=0x7fa850106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa8400089f0 tx=0x7fa84000b040).stop 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 shutdown_connections 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa83c02ff10 0x7fa83c0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 --2- 172.21.15.118:0/4240969930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa850060280 0x7fa850106780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 >> 172.21.15.118:0/4240969930 conn(0x7fa8500f0b70 msgr2=0x7fa8500f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 shutdown_connections 2024-02-23T00:24:12.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:12.513+0000 7fa857dd0700 1 -- 172.21.15.118:0/4240969930 wait complete. 2024-02-23T00:24:12.516 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:13.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:12.512606+0000 mon.smithi090 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.118:0/4240969930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:14.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:13.117978+0000 mgr.smithi090.pxxmlz (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:14.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:14.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:15.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:15.118255+0000 mgr.smithi090.pxxmlz (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- 172.21.15.118:0/1247506801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 msgr2=0x7f72c40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 --2- 172.21.15.118:0/1247506801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c40f4be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f72b8009fb0 tx=0x7f72b8009b30).stop 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- 172.21.15.118:0/1247506801 shutdown_connections 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 --2- 172.21.15.118:0/1247506801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c40f4be0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- 172.21.15.118:0/1247506801 >> 172.21.15.118:0/1247506801 conn(0x7f72c40f02e0 msgr2=0x7f72c40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- 172.21.15.118:0/1247506801 shutdown_connections 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- 172.21.15.118:0/1247506801 wait complete. 2024-02-23T00:24:16.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 Processor -- start 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.326+0000 7f72caa6c700 1 -- start start 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72caa6c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72caa6c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4106cc0 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72c8808700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72c8808700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35632/0 (socket says 172.21.15.118:35632) 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72c8808700 1 -- 172.21.15.118:0/3814936567 learned_addr learned my addr 172.21.15.118:0/3814936567 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72c8808700 1 -- 172.21.15.118:0/3814936567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b8005040 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.327+0000 7f72c8808700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f72b8005010 tx=0x7f72b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8009320 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c4106ec0 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72b8013380 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72b8014e50 con 0x7f72c40f47c0 2024-02-23T00:24:16.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c4107350 con 0x7f72c40f47c0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72b80095f0 con 0x7f72c40f47c0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.328+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c4045ad0 con 0x7f72c40f47c0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.329+0000 7f72c17fa700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 0x7f72b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.329+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72b8040fa0 con 0x7f72c40f47c0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.329+0000 7f72c3fff700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 0x7f72b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.329+0000 7f72c3fff700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 0x7f72b40323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f72b00096a0 tx=0x7f72b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:16.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.333+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72b8014420 con 0x7f72c40f47c0 2024-02-23T00:24:16.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.542+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72c4108200 con 0x7f72c40f47c0 2024-02-23T00:24:16.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.543+0000 7f72c17fa700 1 -- 172.21.15.118:0/3814936567 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72b8010230 con 0x7f72c40f47c0 2024-02-23T00:24:16.544 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:16.544 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 msgr2=0x7f72b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 0x7f72b40323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f72b00096a0 tx=0x7f72b0006b40).stop 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 msgr2=0x7f72c41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f72b8005010 tx=0x7f72b800b040).stop 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 shutdown_connections 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72b402ff10 0x7f72b40323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 --2- 172.21.15.118:0/3814936567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72c40f47c0 0x7f72c41067d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.544+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 >> 172.21.15.118:0/3814936567 conn(0x7f72c40f02e0 msgr2=0x7f72c40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.545+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 shutdown_connections 2024-02-23T00:24:16.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:16.545+0000 7f72caa6c700 1 -- 172.21.15.118:0/3814936567 wait complete. 2024-02-23T00:24:16.547 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:17.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:16.543541+0000 mon.smithi090 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.118:0/3814936567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:18.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:17.118593+0000 mgr.smithi090.pxxmlz (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:18.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:18.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:19.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:19.118991+0000 mgr.smithi090.pxxmlz (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:21.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:21.119264+0000 mgr.smithi090.pxxmlz (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:23.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:23.119584+0000 mgr.smithi090.pxxmlz (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:25.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:25.119919+0000 mgr.smithi090.pxxmlz (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:26.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:25.241788+0000 mon.smithi090 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:24:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:25.249882+0000 mon.smithi090 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:24:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:27.120273+0000 mgr.smithi090.pxxmlz (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:28.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- 172.21.15.118:0/2849545601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 msgr2=0x7f2fb80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:28.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 --2- 172.21.15.118:0/2849545601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb80f90f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2fa8009fb0 tx=0x7f2fa8009b30).stop 2024-02-23T00:24:28.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- 172.21.15.118:0/2849545601 shutdown_connections 2024-02-23T00:24:28.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 --2- 172.21.15.118:0/2849545601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb80f90f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- 172.21.15.118:0/2849545601 >> 172.21.15.118:0/2849545601 conn(0x7f2fb80f0b70 msgr2=0x7f2fb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- 172.21.15.118:0/2849545601 shutdown_connections 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- 172.21.15.118:0/2849545601 wait complete. 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 Processor -- start 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.484+0000 7f2fc0380700 1 -- start start 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fc0380700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fc0380700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb8106c70 con 0x7f2fb8060280 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fbe11c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fbe11c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:49096/0 (socket says 172.21.15.118:49096) 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fbe11c700 1 -- 172.21.15.118:0/3096314842 learned_addr learned my addr 172.21.15.118:0/3096314842 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fbe11c700 1 -- 172.21.15.118:0/3096314842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8005040 con 0x7f2fb8060280 2024-02-23T00:24:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fbe11c700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2fa80089f0 tx=0x7f2fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa8009320 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8106e70 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fa8013380 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa8014e20 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.485+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb8107300 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.486+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb8045ad0 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.486+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fa80095f0 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.486+0000 7f2faeffd700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 0x7f2fa40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.486+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fa8040ef0 con 0x7f2fb8060280 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:28.078 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.487+0000 7f2fbd91b700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 0x7f2fa40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.487+0000 7f2fbd91b700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 0x7f2fa40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2fb40096a0 tx=0x7f2fb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.491+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fa8014420 con 0x7f2fb8060280 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.695+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fb8108150 con 0x7f2fb8060280 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.696+0000 7f2faeffd700 1 -- 172.21.15.118:0/3096314842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fa8010230 con 0x7f2fb8060280 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.697+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 msgr2=0x7f2fa40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.697+0000 7f2fc0380700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 0x7f2fa40323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2fb40096a0 tx=0x7f2fb4006b40).stop 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.697+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 msgr2=0x7f2fb8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:28.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.697+0000 7f2fc0380700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2fa80089f0 tx=0x7f2fa800b040).stop 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 shutdown_connections 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2fa402ff10 0x7f2fa40323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 --2- 172.21.15.118:0/3096314842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fb8060280 0x7f2fb8106780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 >> 172.21.15.118:0/3096314842 conn(0x7f2fb80f0b70 msgr2=0x7f2fb80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 shutdown_connections 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:27.698+0000 7f2fc0380700 1 -- 172.21.15.118:0/3096314842 wait complete. 2024-02-23T00:24:28.080 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:28.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:27.696567+0000 mon.smithi090 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.118:0/3096314842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:29.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:29.120520+0000 mgr.smithi090.pxxmlz (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:31.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:31.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:31.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:31.120794+0000 mgr.smithi090.pxxmlz (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:33.121009+0000 mgr.smithi090.pxxmlz (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:33.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- 172.21.15.118:0/793161403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 msgr2=0x7f3d380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 --2- 172.21.15.118:0/793161403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d380f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3d2c009fb0 tx=0x7f3d2c009b30).stop 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- 172.21.15.118:0/793161403 shutdown_connections 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 --2- 172.21.15.118:0/793161403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d380f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- 172.21.15.118:0/793161403 >> 172.21.15.118:0/793161403 conn(0x7f3d380f09a0 msgr2=0x7f3d380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- 172.21.15.118:0/793161403 shutdown_connections 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- 172.21.15.118:0/793161403 wait complete. 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 Processor -- start 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 -- start start 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.687+0000 7f3d3df86700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:33.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3df86700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d381059c0 con 0x7f3d380f51a0 2024-02-23T00:24:33.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3cf84700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:33.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3cf84700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:49126/0 (socket says 172.21.15.118:49126) 2024-02-23T00:24:33.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3cf84700 1 -- 172.21.15.118:0/3235746047 learned_addr learned my addr 172.21.15.118:0/3235746047 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3cf84700 1 -- 172.21.15.118:0/3235746047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c005040 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3cf84700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3d2c0089f0 tx=0x7f3d2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c009320 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38105bc0 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d2c013380 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c014e20 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.688+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38106050 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.689+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d2c0095f0 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.689+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d38045ad0 con 0x7f3d380f51a0 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.689+0000 7f3d357fa700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 0x7f3d280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:33.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.689+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d2c040ef0 con 0x7f3d380f51a0 2024-02-23T00:24:33.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.690+0000 7f3d37fff700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 0x7f3d280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:33.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.690+0000 7f3d37fff700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 0x7f3d280323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3d240096a0 tx=0x7f3d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:33.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.694+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2c014420 con 0x7f3d380f51a0 2024-02-23T00:24:33.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.899+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d38106ea0 con 0x7f3d380f51a0 2024-02-23T00:24:33.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.900+0000 7f3d357fa700 1 -- 172.21.15.118:0/3235746047 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d2c010230 con 0x7f3d380f51a0 2024-02-23T00:24:33.902 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:33.902 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:33.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 msgr2=0x7f3d280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:33.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 0x7f3d280323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3d240096a0 tx=0x7f3d24006b40).stop 2024-02-23T00:24:33.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 msgr2=0x7f3d381054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:33.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3d2c0089f0 tx=0x7f3d2c00b040).stop 2024-02-23T00:24:33.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 shutdown_connections 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d2802ff10 0x7f3d280323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 --2- 172.21.15.118:0/3235746047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d380f51a0 0x7f3d381054d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 >> 172.21.15.118:0/3235746047 conn(0x7f3d380f09a0 msgr2=0x7f3d380f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 shutdown_connections 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:33.902+0000 7f3d3df86700 1 -- 172.21.15.118:0/3235746047 wait complete. 2024-02-23T00:24:33.904 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:34.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:33.901008+0000 mon.smithi090 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.118:0/3235746047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:35.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:35.121238+0000 mgr.smithi090.pxxmlz (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:35.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:35.598 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3202628912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 msgr2=0x7fe0e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3202628912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e40f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe0d8009fb0 tx=0x7fe0d8009b30).stop 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3202628912 shutdown_connections 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3202628912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e40f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3202628912 >> 172.21.15.118:0/3202628912 conn(0x7fe0e40f1c70 msgr2=0x7fe0e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3202628912 shutdown_connections 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.456+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3202628912 wait complete. 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0ea5c0700 1 Processor -- start 2024-02-23T00:24:37.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0ea5c0700 1 -- start start 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0ea5c0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0ea5c0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4106cd0 con 0x7fe0e40f6510 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0e3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0e3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:45968/0 (socket says 172.21.15.118:45968) 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.457+0000 7fe0e3fff700 1 -- 172.21.15.118:0/3031555577 learned_addr learned my addr 172.21.15.118:0/3031555577 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0e3fff700 1 -- 172.21.15.118:0/3031555577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8005040 con 0x7fe0e40f6510 2024-02-23T00:24:37.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0e3fff700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe0d8005010 tx=0x7fe0d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:37.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0d8009320 con 0x7fe0e40f6510 2024-02-23T00:24:37.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e4106ed0 con 0x7fe0e40f6510 2024-02-23T00:24:37.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0d8013380 con 0x7fe0e40f6510 2024-02-23T00:24:37.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0d8014e50 con 0x7fe0e40f6510 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.458+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e4107360 con 0x7fe0e40f6510 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.459+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0d80263a0 con 0x7fe0e40f6510 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.459+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4045ad0 con 0x7fe0e40f6510 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.459+0000 7fe0e0ff9700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 0x7fe0cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.459+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0d8040f90 con 0x7fe0e40f6510 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.459+0000 7fe0e37fe700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 0x7fe0cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:37.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.460+0000 7fe0e37fe700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 0x7fe0cc0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe0d40096a0 tx=0x7fe0d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:37.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.464+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0d80126f0 con 0x7fe0e40f6510 2024-02-23T00:24:37.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:37.121491+0000 mgr.smithi090.pxxmlz (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:37.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.665+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0e4108210 con 0x7fe0e40f6510 2024-02-23T00:24:37.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.666+0000 7fe0e0ff9700 1 -- 172.21.15.118:0/3031555577 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0d800f340 con 0x7fe0e40f6510 2024-02-23T00:24:37.667 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:37.668 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:37.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 msgr2=0x7fe0cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:37.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 0x7fe0cc0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe0d40096a0 tx=0x7fe0d4006b40).stop 2024-02-23T00:24:37.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 msgr2=0x7fe0e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:37.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe0d8005010 tx=0x7fe0d800b040).stop 2024-02-23T00:24:37.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 shutdown_connections 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe0cc02ff10 0x7fe0cc0323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 --2- 172.21.15.118:0/3031555577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e40f6510 0x7fe0e41067e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 >> 172.21.15.118:0/3031555577 conn(0x7fe0e40f1c70 msgr2=0x7fe0e40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 shutdown_connections 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:37.668+0000 7fe0ea5c0700 1 -- 172.21.15.118:0/3031555577 wait complete. 2024-02-23T00:24:37.670 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:38.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:37.666951+0000 mon.smithi090 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.118:0/3031555577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:39.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:39.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:39.121692+0000 mgr.smithi090.pxxmlz (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 -- 172.21.15.118:0/1210273085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 msgr2=0x7fc0a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/1210273085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a00f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc088009fb0 tx=0x7fc088009b30).stop 2024-02-23T00:24:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 -- 172.21.15.118:0/1210273085 shutdown_connections 2024-02-23T00:24:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/1210273085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a00f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 -- 172.21.15.118:0/1210273085 >> 172.21.15.118:0/1210273085 conn(0x7fc0a00f0b70 msgr2=0x7fc0a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 -- 172.21.15.118:0/1210273085 shutdown_connections 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.461+0000 7fc0a4f34700 1 -- 172.21.15.118:0/1210273085 wait complete. 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc0a4f34700 1 Processor -- start 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc0a4f34700 1 -- start start 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc0a4f34700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc0a4f34700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0106cd0 con 0x7fc0a00f5410 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc09e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc09e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:45978/0 (socket says 172.21.15.118:45978) 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.462+0000 7fc09e59c700 1 -- 172.21.15.118:0/3917420758 learned_addr learned my addr 172.21.15.118:0/3917420758 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc09e59c700 1 -- 172.21.15.118:0/3917420758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088005040 con 0x7fc0a00f5410 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc09e59c700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc088005010 tx=0x7fc08800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc088009320 con 0x7fc0a00f5410 2024-02-23T00:24:41.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc088013380 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc088014e50 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a0106ed0 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.463+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a0107360 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.464+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0880095f0 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.464+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a0045ad0 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.464+0000 7fc096ffd700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 0x7fc08c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.464+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc088040e60 con 0x7fc0a00f5410 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.464+0000 7fc09dd9b700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 0x7fc08c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.465+0000 7fc09dd9b700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 0x7fc08c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:41.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.468+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc088014420 con 0x7fc0a00f5410 2024-02-23T00:24:41.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.677+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0a0108210 con 0x7fc0a00f5410 2024-02-23T00:24:41.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.677+0000 7fc096ffd700 1 -- 172.21.15.118:0/3917420758 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc08801d980 con 0x7fc0a00f5410 2024-02-23T00:24:41.680 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:41.680 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 msgr2=0x7fc08c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 0x7fc08c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc0900096a0 tx=0x7fc090006b40).stop 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 msgr2=0x7fc0a01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc088005010 tx=0x7fc08800b040).stop 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 shutdown_connections 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc08c02ff10 0x7fc08c0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 --2- 172.21.15.118:0/3917420758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0a00f5410 0x7fc0a01067e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 >> 172.21.15.118:0/3917420758 conn(0x7fc0a00f0b70 msgr2=0x7fc0a00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:41.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 shutdown_connections 2024-02-23T00:24:41.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:41.679+0000 7fc0a4f34700 1 -- 172.21.15.118:0/3917420758 wait complete. 2024-02-23T00:24:41.682 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:41.122034+0000 mgr.smithi090.pxxmlz (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:42.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:41.678272+0000 mon.smithi090 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.118:0/3917420758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:43.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:43.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:43.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:43.122362+0000 mgr.smithi090.pxxmlz (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.357+0000 7f82b0b8c700 1 -- 172.21.15.118:0/4005687110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 msgr2=0x7f82ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.357+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/4005687110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac0f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f82a0009fb0 tx=0x7f82a0009b30).stop 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.357+0000 7f82b0b8c700 1 -- 172.21.15.118:0/4005687110 shutdown_connections 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.357+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/4005687110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac0f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.357+0000 7f82b0b8c700 1 -- 172.21.15.118:0/4005687110 >> 172.21.15.118:0/4005687110 conn(0x7f82ac0f0b70 msgr2=0x7f82ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:45.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 -- 172.21.15.118:0/4005687110 shutdown_connections 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 -- 172.21.15.118:0/4005687110 wait complete. 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 Processor -- start 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 -- start start 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.358+0000 7f82b0b8c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac060240 con 0x7f82ac0f5410 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54330/0 (socket says 172.21.15.118:54330) 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82aa59c700 1 -- 172.21.15.118:0/685474485 learned_addr learned my addr 172.21.15.118:0/685474485 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82aa59c700 1 -- 172.21.15.118:0/685474485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0005040 con 0x7f82ac0f5410 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82aa59c700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f82a00089f0 tx=0x7f82a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:45.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a0009320 con 0x7f82ac0f5410 2024-02-23T00:24:45.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac060440 con 0x7f82ac0f5410 2024-02-23T00:24:45.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82a0013380 con 0x7f82ac0f5410 2024-02-23T00:24:45.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a001ed80 con 0x7f82ac0f5410 2024-02-23T00:24:45.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.359+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac109bc0 con 0x7f82ac0f5410 2024-02-23T00:24:45.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.360+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82a0014420 con 0x7f82ac0f5410 2024-02-23T00:24:45.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.360+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac045ad0 con 0x7f82ac0f5410 2024-02-23T00:24:45.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.360+0000 7f829affd700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 0x7f8294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:45.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.360+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82a00410a0 con 0x7f82ac0f5410 2024-02-23T00:24:45.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.360+0000 7f82a9d9b700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 0x7f8294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:45.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.361+0000 7f82a9d9b700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 0x7f8294032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:45.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.365+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82a00095f0 con 0x7f82ac0f5410 2024-02-23T00:24:45.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:45.122679+0000 mgr.smithi090.pxxmlz (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:45.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.570+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82ac10a3d0 con 0x7f82ac0f5410 2024-02-23T00:24:45.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.570+0000 7f829affd700 1 -- 172.21.15.118:0/685474485 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82a001cb10 con 0x7f82ac0f5410 2024-02-23T00:24:45.572 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:45.572 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:45.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 msgr2=0x7f8294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:45.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 0x7f8294032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c006b40).stop 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 msgr2=0x7f82ac109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f82a00089f0 tx=0x7f82a000b040).stop 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 shutdown_connections 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f829402ff60 0x7f8294032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 --2- 172.21.15.118:0/685474485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82ac0f5410 0x7f82ac109510 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 >> 172.21.15.118:0/685474485 conn(0x7f82ac0f0b70 msgr2=0x7f82ac0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 shutdown_connections 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:45.572+0000 7f82b0b8c700 1 -- 172.21.15.118:0/685474485 wait complete. 2024-02-23T00:24:45.574 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:46.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:45.571428+0000 mon.smithi090 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.118:0/685474485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:47.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:47.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:47.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:47.122908+0000 mgr.smithi090.pxxmlz (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- 172.21.15.118:0/2411201526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 msgr2=0x7ff46c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 --2- 172.21.15.118:0/2411201526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff45c009fb0 tx=0x7ff45c009b30).stop 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- 172.21.15.118:0/2411201526 shutdown_connections 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 --2- 172.21.15.118:0/2411201526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- 172.21.15.118:0/2411201526 >> 172.21.15.118:0/2411201526 conn(0x7ff46c0f0b70 msgr2=0x7ff46c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- 172.21.15.118:0/2411201526 shutdown_connections 2024-02-23T00:24:49.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- 172.21.15.118:0/2411201526 wait complete. 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 Processor -- start 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.187+0000 7ff4725f4700 1 -- start start 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff4725f4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff4725f4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0fb960 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff46bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff46bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54352/0 (socket says 172.21.15.118:54352) 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff46bfff700 1 -- 172.21.15.118:0/3428040939 learned_addr learned my addr 172.21.15.118:0/3428040939 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff46bfff700 1 -- 172.21.15.118:0/3428040939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c005040 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff46bfff700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff45c005010 tx=0x7ff45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c009320 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c0fbb00 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff45c013380 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c014e50 con 0x7ff46c0f5410 2024-02-23T00:24:49.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.188+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c0fbf90 con 0x7ff46c0f5410 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.189+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff45c014420 con 0x7ff46c0f5410 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.189+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c059100 con 0x7ff46c0f5410 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.189+0000 7ff468ff9700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 0x7ff4540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.189+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff45c040e30 con 0x7ff46c0f5410 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.190+0000 7ff46b7fe700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 0x7ff4540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.190+0000 7ff46b7fe700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 0x7ff4540323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff4600096a0 tx=0x7ff460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:49.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.194+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff45c026bf0 con 0x7ff46c0f5410 2024-02-23T00:24:49.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.401+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff46c0fafa0 con 0x7ff46c0f5410 2024-02-23T00:24:49.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.402+0000 7ff468ff9700 1 -- 172.21.15.118:0/3428040939 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff45c01db70 con 0x7ff46c0f5410 2024-02-23T00:24:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:49.404 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 msgr2=0x7ff4540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 0x7ff4540323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff4600096a0 tx=0x7ff460006b40).stop 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 msgr2=0x7ff46c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff45c005010 tx=0x7ff45c00b040).stop 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 shutdown_connections 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff45402ff10 0x7ff4540323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 --2- 172.21.15.118:0/3428040939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46c0f5410 0x7ff46c0fd4e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 >> 172.21.15.118:0/3428040939 conn(0x7ff46c0f0b70 msgr2=0x7ff46c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 shutdown_connections 2024-02-23T00:24:49.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:49.403+0000 7ff4725f4700 1 -- 172.21.15.118:0/3428040939 wait complete. 2024-02-23T00:24:49.407 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:49.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:49.123184+0000 mgr.smithi090.pxxmlz (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:50.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:49.403463+0000 mon.smithi090 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.118:0/3428040939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:51.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:51.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:51.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:51.123537+0000 mgr.smithi090.pxxmlz (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- 172.21.15.118:0/756836244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 msgr2=0x7fe11c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 --2- 172.21.15.118:0/756836244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c0f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe118009fb0 tx=0x7fe118009b30).stop 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- 172.21.15.118:0/756836244 shutdown_connections 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 --2- 172.21.15.118:0/756836244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c0f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- 172.21.15.118:0/756836244 >> 172.21.15.118:0/756836244 conn(0x7fe11c0f1c70 msgr2=0x7fe11c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- 172.21.15.118:0/756836244 shutdown_connections 2024-02-23T00:24:52.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- 172.21.15.118:0/756836244 wait complete. 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 Processor -- start 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- start start 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.975+0000 7fe12430a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c069e20 con 0x7fe11c0f6510 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe1220a6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe1220a6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54360/0 (socket says 172.21.15.118:54360) 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe1220a6700 1 -- 172.21.15.118:0/2664798423 learned_addr learned my addr 172.21.15.118:0/2664798423 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe1220a6700 1 -- 172.21.15.118:0/2664798423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118005040 con 0x7fe11c0f6510 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe1220a6700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe118005010 tx=0x7fe11800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe118009320 con 0x7fe11c0f6510 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c069fc0 con 0x7fe11c0f6510 2024-02-23T00:24:52.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe118013380 con 0x7fe11c0f6510 2024-02-23T00:24:52.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe118014e50 con 0x7fe11c0f6510 2024-02-23T00:24:52.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.976+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c06a450 con 0x7fe11c0f6510 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.977+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1180263a0 con 0x7fe11c0f6510 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.977+0000 7fe112ffd700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 0x7fe1080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.977+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe118040e70 con 0x7fe11c0f6510 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.977+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c059100 con 0x7fe11c0f6510 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.978+0000 7fe1218a5700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 0x7fe1080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:52.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.978+0000 7fe1218a5700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 0x7fe1080323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe10c0096a0 tx=0x7fe10c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:52.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:52.983+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe118012920 con 0x7fe11c0f6510 2024-02-23T00:24:53.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.189+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe11c06ab50 con 0x7fe11c0f6510 2024-02-23T00:24:53.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.190+0000 7fe112ffd700 1 -- 172.21.15.118:0/2664798423 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe11802b020 con 0x7fe11c0f6510 2024-02-23T00:24:53.192 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:53.192 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 msgr2=0x7fe1080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 0x7fe1080323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe10c0096a0 tx=0x7fe10c006b40).stop 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 msgr2=0x7fe11c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe118005010 tx=0x7fe11800b040).stop 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 shutdown_connections 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe10802ff10 0x7fe1080323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 --2- 172.21.15.118:0/2664798423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe11c0f6510 0x7fe11c06b9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:53.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 >> 172.21.15.118:0/2664798423 conn(0x7fe11c0f1c70 msgr2=0x7fe11c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:53.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 shutdown_connections 2024-02-23T00:24:53.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:53.191+0000 7fe12430a700 1 -- 172.21.15.118:0/2664798423 wait complete. 2024-02-23T00:24:53.195 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:53.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:53.123856+0000 mgr.smithi090.pxxmlz (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:53.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:53.191524+0000 mon.smithi090 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.118:0/2664798423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:54.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:24:54.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:24:55.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:55.124167+0000 mgr.smithi090.pxxmlz (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:57.124497+0000 mgr.smithi090.pxxmlz (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:58.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:24:58.030652+0000 mon.smithi090 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.118:0/595191842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 -- 172.21.15.118:0/72977703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 msgr2=0x7f0bc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 --2- 172.21.15.118:0/72977703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc00f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0bbc009fb0 tx=0x7f0bbc009b30).stop 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 -- 172.21.15.118:0/72977703 shutdown_connections 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 --2- 172.21.15.118:0/72977703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc00f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 -- 172.21.15.118:0/72977703 >> 172.21.15.118:0/72977703 conn(0x7f0bc00f0b70 msgr2=0x7f0bc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 -- 172.21.15.118:0/72977703 shutdown_connections 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.815+0000 7f0bc8778700 1 -- 172.21.15.118:0/72977703 wait complete. 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc8778700 1 Processor -- start 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc8778700 1 -- start start 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc8778700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc8778700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0105c10 con 0x7f0bc00f5410 2024-02-23T00:24:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc6514700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc6514700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47762/0 (socket says 172.21.15.118:47762) 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.816+0000 7f0bc6514700 1 -- 172.21.15.118:0/595191842 learned_addr learned my addr 172.21.15.118:0/595191842 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bc6514700 1 -- 172.21.15.118:0/595191842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc005040 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bc6514700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0bbc0089f0 tx=0x7f0bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bbc009320 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc0105e10 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bbc013380 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bbc01ed80 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.817+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc01062a0 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bbc01e420 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0045ad0 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bb6ffd700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 0x7f0bac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bbc040e70 con 0x7f0bc00f5410 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bc5d13700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 0x7f0bac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.818+0000 7f0bc5d13700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 0x7f0bac032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0bb00096a0 tx=0x7f0bb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:57.822+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bbc00e2a0 con 0x7f0bc00f5410 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.029+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bc0107150 con 0x7f0bc00f5410 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.030+0000 7f0bb6ffd700 1 -- 172.21.15.118:0/595191842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bbc00e2a0 con 0x7f0bc00f5410 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 msgr2=0x7f0bac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 0x7f0bac032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0bb00096a0 tx=0x7f0bb0006b40).stop 2024-02-23T00:24:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 msgr2=0x7f0bc0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0bbc0089f0 tx=0x7f0bbc00b040).stop 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 shutdown_connections 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bac02ff60 0x7f0bac032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 --2- 172.21.15.118:0/595191842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f5410 0x7f0bc0105720 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 >> 172.21.15.118:0/595191842 conn(0x7f0bc00f0b70 msgr2=0x7f0bc00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 shutdown_connections 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:24:58.031+0000 7f0bc8778700 1 -- 172.21.15.118:0/595191842 wait complete. 2024-02-23T00:24:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:24:59.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:24:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:24:59.124825+0000 mgr.smithi090.pxxmlz (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:00.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:00.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:01.125141+0000 mgr.smithi090.pxxmlz (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:02.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 -- 172.21.15.118:0/1505574214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 msgr2=0x7f98140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:02.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 --2- 172.21.15.118:0/1505574214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98140f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9804009fb0 tx=0x7f9804009b30).stop 2024-02-23T00:25:02.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 -- 172.21.15.118:0/1505574214 shutdown_connections 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 --2- 172.21.15.118:0/1505574214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98140f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 -- 172.21.15.118:0/1505574214 >> 172.21.15.118:0/1505574214 conn(0x7f98140f1c70 msgr2=0x7f98140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 -- 172.21.15.118:0/1505574214 shutdown_connections 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.400+0000 7f981a931700 1 -- 172.21.15.118:0/1505574214 wait complete. 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f981a931700 1 Processor -- start 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f981a931700 1 -- start start 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f981a931700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f981a931700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9814106cd0 con 0x7f98140f6510 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f9813fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f9813fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47776/0 (socket says 172.21.15.118:47776) 2024-02-23T00:25:02.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.401+0000 7f9813fff700 1 -- 172.21.15.118:0/2796879674 learned_addr learned my addr 172.21.15.118:0/2796879674 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f9813fff700 1 -- 172.21.15.118:0/2796879674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9804005040 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f9813fff700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f98040089f0 tx=0x7f980400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9804009320 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9804013380 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9814106ed0 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980401ed80 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.402+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9814107360 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f980401e420 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9814045ad0 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f9810ff9700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 0x7f97fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9804040e70 con 0x7f98140f6510 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f98137fe700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 0x7f97fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:02.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.403+0000 7f98137fe700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 0x7f97fc032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f98080096a0 tx=0x7f9808006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:02.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.408+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f980400e2a0 con 0x7f98140f6510 2024-02-23T00:25:02.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.611+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9814108210 con 0x7f98140f6510 2024-02-23T00:25:02.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.612+0000 7f9810ff9700 1 -- 172.21.15.118:0/2796879674 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f980400e2a0 con 0x7f98140f6510 2024-02-23T00:25:02.614 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:02.614 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 msgr2=0x7f97fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 0x7f97fc032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f98080096a0 tx=0x7f9808006b40).stop 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 msgr2=0x7f98141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f98040089f0 tx=0x7f980400b040).stop 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 shutdown_connections 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f97fc02ff60 0x7f97fc032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 --2- 172.21.15.118:0/2796879674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98140f6510 0x7f98141067e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.613+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 >> 172.21.15.118:0/2796879674 conn(0x7f98140f1c70 msgr2=0x7f98140f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:02.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.614+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 shutdown_connections 2024-02-23T00:25:02.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:02.614+0000 7f981a931700 1 -- 172.21.15.118:0/2796879674 wait complete. 2024-02-23T00:25:02.616 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:03.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:02.612868+0000 mon.smithi090 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.118:0/2796879674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:04.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:03.125404+0000 mgr.smithi090.pxxmlz (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:04.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:04.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:05.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:05.125750+0000 mgr.smithi090.pxxmlz (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:06.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- 172.21.15.118:0/130050150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 msgr2=0x7fc4800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 --2- 172.21.15.118:0/130050150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4800f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fc470009fb0 tx=0x7fc470009b30).stop 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- 172.21.15.118:0/130050150 shutdown_connections 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 --2- 172.21.15.118:0/130050150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4800f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- 172.21.15.118:0/130050150 >> 172.21.15.118:0/130050150 conn(0x7fc4800f1c70 msgr2=0x7fc4800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- 172.21.15.118:0/130050150 shutdown_connections 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- 172.21.15.118:0/130050150 wait complete. 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 Processor -- start 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.017+0000 7fc4868c9700 1 -- start start 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc4868c9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc4868c9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc480106cd0 con 0x7fc4800f6510 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41104/0 (socket says 172.21.15.118:41104) 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47ffff700 1 -- 172.21.15.118:0/4206126907 learned_addr learned my addr 172.21.15.118:0/4206126907 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47ffff700 1 -- 172.21.15.118:0/4206126907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470005040 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47ffff700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc470005010 tx=0x7fc47000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc470009320 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc480106ed0 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc470013380 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc470014e50 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.018+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc480107360 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.019+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4700263a0 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.019+0000 7fc47cff9700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 0x7fc4680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.019+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc470040fb0 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.019+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480045ad0 con 0x7fc4800f6510 2024-02-23T00:25:06.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.019+0000 7fc47f7fe700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 0x7fc4680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:06.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.020+0000 7fc47f7fe700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 0x7fc4680323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc4740096a0 tx=0x7fc474006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:06.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.024+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc470012920 con 0x7fc4800f6510 2024-02-23T00:25:06.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.226+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc480108080 con 0x7fc4800f6510 2024-02-23T00:25:06.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.227+0000 7fc47cff9700 1 -- 172.21.15.118:0/4206126907 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc47002b020 con 0x7fc4800f6510 2024-02-23T00:25:06.229 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:06.229 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:06.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 msgr2=0x7fc4680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:06.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 0x7fc4680323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc4740096a0 tx=0x7fc474006b40).stop 2024-02-23T00:25:06.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 msgr2=0x7fc4801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:06.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc470005010 tx=0x7fc47000b040).stop 2024-02-23T00:25:06.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 shutdown_connections 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc46802ff10 0x7fc4680323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 --2- 172.21.15.118:0/4206126907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4800f6510 0x7fc4801067e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 >> 172.21.15.118:0/4206126907 conn(0x7fc4800f1c70 msgr2=0x7fc4800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 shutdown_connections 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:06.229+0000 7fc4868c9700 1 -- 172.21.15.118:0/4206126907 wait complete. 2024-02-23T00:25:06.232 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:06.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:06.228178+0000 mon.smithi090 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.118:0/4206126907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:07.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:07.126097+0000 mgr.smithi090.pxxmlz (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:08.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:08.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:09.126362+0000 mgr.smithi090.pxxmlz (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- 172.21.15.118:0/3776926138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 msgr2=0x7f1da40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/3776926138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1da0009fb0 tx=0x7f1da0009b30).stop 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- 172.21.15.118:0/3776926138 shutdown_connections 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/3776926138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- 172.21.15.118:0/3776926138 >> 172.21.15.118:0/3776926138 conn(0x7f1da40f09a0 msgr2=0x7f1da40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- 172.21.15.118:0/3776926138 shutdown_connections 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- 172.21.15.118:0/3776926138 wait complete. 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 Processor -- start 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.028+0000 7f1daa5dd700 1 -- start start 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1daa5dd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1daa5dd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40f13a0 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1da95db700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1da95db700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41112/0 (socket says 172.21.15.118:41112) 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1da95db700 1 -- 172.21.15.118:0/211900783 learned_addr learned my addr 172.21.15.118:0/211900783 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1da95db700 1 -- 172.21.15.118:0/211900783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0005040 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1da95db700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f1da00089f0 tx=0x7f1da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0009320 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da40f1540 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1da0013380 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0014e20 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.029+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da40f19d0 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.030+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4059100 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.030+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1da00095f0 con 0x7f1da40f51a0 2024-02-23T00:25:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.031+0000 7f1d99ffb700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 0x7f1d900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.031+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1da0040e90 con 0x7f1da40f51a0 2024-02-23T00:25:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.031+0000 7f1da8dda700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 0x7f1d900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.031+0000 7f1da8dda700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 0x7f1d900323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1d940096a0 tx=0x7f1d94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.035+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1da0014420 con 0x7f1da40f51a0 2024-02-23T00:25:10.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.241+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1da4045ad0 con 0x7f1da40f51a0 2024-02-23T00:25:10.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.242+0000 7f1d99ffb700 1 -- 172.21.15.118:0/211900783 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1da001d980 con 0x7f1da40f51a0 2024-02-23T00:25:10.244 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:10.244 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:10.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 msgr2=0x7f1d900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:10.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 0x7f1d900323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1d940096a0 tx=0x7f1d94006b40).stop 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 msgr2=0x7f1da40f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f1da00089f0 tx=0x7f1da000b040).stop 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 shutdown_connections 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1d9002ff10 0x7f1d900323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 --2- 172.21.15.118:0/211900783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da40f51a0 0x7f1da40f2f20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 >> 172.21.15.118:0/211900783 conn(0x7f1da40f09a0 msgr2=0x7f1da405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 shutdown_connections 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:10.244+0000 7f1daa5dd700 1 -- 172.21.15.118:0/211900783 wait complete. 2024-02-23T00:25:10.246 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:10.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:10.242789+0000 mon.smithi090 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.118:0/211900783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:11.126696+0000 mgr.smithi090.pxxmlz (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:12.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:12.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:13.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:13.126990+0000 mgr.smithi090.pxxmlz (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 -- 172.21.15.118:0/3383213856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 msgr2=0x7f01e00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 --2- 172.21.15.118:0/3383213856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e00f90f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f01d0009fb0 tx=0x7f01d0009b30).stop 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 -- 172.21.15.118:0/3383213856 shutdown_connections 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 --2- 172.21.15.118:0/3383213856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e00f90f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 -- 172.21.15.118:0/3383213856 >> 172.21.15.118:0/3383213856 conn(0x7f01e00f0b70 msgr2=0x7f01e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 -- 172.21.15.118:0/3383213856 shutdown_connections 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.976+0000 7f01e7645700 1 -- 172.21.15.118:0/3383213856 wait complete. 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e7645700 1 Processor -- start 2024-02-23T00:25:13.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e7645700 1 -- start start 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e7645700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e7645700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0106c70 con 0x7f01e0060280 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e53e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e53e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41136/0 (socket says 172.21.15.118:41136) 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.977+0000 7f01e53e1700 1 -- 172.21.15.118:0/3067344310 learned_addr learned my addr 172.21.15.118:0/3067344310 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01e53e1700 1 -- 172.21.15.118:0/3067344310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0005040 con 0x7f01e0060280 2024-02-23T00:25:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01e53e1700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f01d00089f0 tx=0x7f01d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d0009320 con 0x7f01e0060280 2024-02-23T00:25:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e0106e70 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01d0013380 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d0014e20 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.978+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e0107300 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.979+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01d00095f0 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.979+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e0045ad0 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.979+0000 7f01d5ffb700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.979+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01d0040ed0 con 0x7f01e0060280 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.979+0000 7f01e4be0700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.980+0000 7f01e4be0700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 0x7f01cc0323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f01dc0096a0 tx=0x7f01dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:13.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:13.984+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01d0014420 con 0x7f01e0060280 2024-02-23T00:25:14.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.190+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e01081b0 con 0x7f01e0060280 2024-02-23T00:25:14.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.191+0000 7f01d5ffb700 1 -- 172.21.15.118:0/3067344310 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01d001d980 con 0x7f01e0060280 2024-02-23T00:25:14.192 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:14.193 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:14.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 msgr2=0x7f01cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:14.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 0x7f01cc0323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f01dc0096a0 tx=0x7f01dc006b40).stop 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 msgr2=0x7f01e0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f01d00089f0 tx=0x7f01d000b040).stop 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 shutdown_connections 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01cc02ff10 0x7f01cc0323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 --2- 172.21.15.118:0/3067344310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01e0060280 0x7f01e0106780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 >> 172.21.15.118:0/3067344310 conn(0x7f01e00f0b70 msgr2=0x7f01e00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 shutdown_connections 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:14.193+0000 7f01e7645700 1 -- 172.21.15.118:0/3067344310 wait complete. 2024-02-23T00:25:14.195 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:14.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:14.192100+0000 mon.smithi090 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.118:0/3067344310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:15.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:15.127350+0000 mgr.smithi090.pxxmlz (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:15.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:15.955 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:17.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:17.127699+0000 mgr.smithi090.pxxmlz (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:17.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.779+0000 7f283fb18700 1 -- 172.21.15.118:0/3549309231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 msgr2=0x7f28380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:17.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.779+0000 7f283fb18700 1 --2- 172.21.15.118:0/3549309231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f28380f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2828009fb0 tx=0x7f2828009b30).stop 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.779+0000 7f283fb18700 1 -- 172.21.15.118:0/3549309231 shutdown_connections 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.779+0000 7f283fb18700 1 --2- 172.21.15.118:0/3549309231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f28380f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.779+0000 7f283fb18700 1 -- 172.21.15.118:0/3549309231 >> 172.21.15.118:0/3549309231 conn(0x7f28380f0b70 msgr2=0x7f28380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 -- 172.21.15.118:0/3549309231 shutdown_connections 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 -- 172.21.15.118:0/3549309231 wait complete. 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 Processor -- start 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 -- start start 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283fb18700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2838105c10 con 0x7f28380f5410 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283d8b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283d8b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50160/0 (socket says 172.21.15.118:50160) 2024-02-23T00:25:17.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.780+0000 7f283d8b4700 1 -- 172.21.15.118:0/3849889833 learned_addr learned my addr 172.21.15.118:0/3849889833 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f283d8b4700 1 -- 172.21.15.118:0/3849889833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828005040 con 0x7f28380f5410 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f283d8b4700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2828005010 tx=0x7f282800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828009320 con 0x7f28380f5410 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2838105e10 con 0x7f28380f5410 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2828013380 con 0x7f28380f5410 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2828014e50 con 0x7f28380f5410 2024-02-23T00:25:17.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.781+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28381062a0 con 0x7f28380f5410 2024-02-23T00:25:17.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.782+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28280095f0 con 0x7f28380f5410 2024-02-23T00:25:17.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.782+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2838045ad0 con 0x7f28380f5410 2024-02-23T00:25:17.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.782+0000 7f282e7fc700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 0x7f28240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:17.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.782+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2828040e70 con 0x7f28380f5410 2024-02-23T00:25:17.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.782+0000 7f283d0b3700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 0x7f28240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:17.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.783+0000 7f283d0b3700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 0x7f28240323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f28340096a0 tx=0x7f2834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:17.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.787+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2828014420 con 0x7f28380f5410 2024-02-23T00:25:17.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.993+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2838107150 con 0x7f28380f5410 2024-02-23T00:25:17.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.993+0000 7f282e7fc700 1 -- 172.21.15.118:0/3849889833 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2828010230 con 0x7f28380f5410 2024-02-23T00:25:17.995 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:17.996 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 msgr2=0x7f28240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 0x7f28240323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f28340096a0 tx=0x7f2834006b40).stop 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 msgr2=0x7f2838105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2828005010 tx=0x7f282800b040).stop 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 shutdown_connections 2024-02-23T00:25:17.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f282402ff10 0x7f28240323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:17.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 --2- 172.21.15.118:0/3849889833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28380f5410 0x7f2838105720 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:17.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 >> 172.21.15.118:0/3849889833 conn(0x7f28380f0b70 msgr2=0x7f28380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:17.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 shutdown_connections 2024-02-23T00:25:17.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:17.995+0000 7f283fb18700 1 -- 172.21.15.118:0/3849889833 wait complete. 2024-02-23T00:25:17.998 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:18.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:17.994128+0000 mon.smithi090 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.118:0/3849889833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:19.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:19.127985+0000 mgr.smithi090.pxxmlz (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:19.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:19.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:21.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 -- 172.21.15.118:0/2853016890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 msgr2=0x7f4bc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 --2- 172.21.15.118:0/2853016890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc00f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f4bbc009fb0 tx=0x7f4bbc009b30).stop 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 -- 172.21.15.118:0/2853016890 shutdown_connections 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 --2- 172.21.15.118:0/2853016890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc00f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 -- 172.21.15.118:0/2853016890 >> 172.21.15.118:0/2853016890 conn(0x7f4bc00f09a0 msgr2=0x7f4bc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 -- 172.21.15.118:0/2853016890 shutdown_connections 2024-02-23T00:25:21.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 -- 172.21.15.118:0/2853016890 wait complete. 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.599+0000 7f4bc6826700 1 Processor -- start 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc6826700 1 -- start start 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc6826700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc6826700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0060280 con 0x7f4bc00f51a0 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc5824700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc5824700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50186/0 (socket says 172.21.15.118:50186) 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc5824700 1 -- 172.21.15.118:0/380656438 learned_addr learned my addr 172.21.15.118:0/380656438 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc5824700 1 -- 172.21.15.118:0/380656438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc005040 con 0x7f4bc00f51a0 2024-02-23T00:25:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc5824700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4bbc0089f0 tx=0x7f4bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc009320 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bbc013380 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc0060480 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bbc014e20 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.600+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc01099c0 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.601+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bbc014420 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.601+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc0045ad0 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.601+0000 7f4bb67fc700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 0x7f4bac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.601+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bbc040ff0 con 0x7f4bc00f51a0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.602+0000 7f4bc5023700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 0x7f4bac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:21.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.602+0000 7f4bc5023700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 0x7f4bac0323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4bb00096a0 tx=0x7f4bb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:21.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.606+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bbc0095f0 con 0x7f4bc00f51a0 2024-02-23T00:25:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:21.128225+0000 mgr.smithi090.pxxmlz (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:21.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.815+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bc00fad30 con 0x7f4bc00f51a0 2024-02-23T00:25:21.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.816+0000 7f4bb67fc700 1 -- 172.21.15.118:0/380656438 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bbc01db70 con 0x7f4bc00f51a0 2024-02-23T00:25:21.817 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:21.818 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.817+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 msgr2=0x7f4bac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.817+0000 7f4bc6826700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 0x7f4bac0323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4bb00096a0 tx=0x7f4bb0006b40).stop 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.817+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 msgr2=0x7f4bc0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.817+0000 7f4bc6826700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4bbc0089f0 tx=0x7f4bbc00b040).stop 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.817+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 shutdown_connections 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.818+0000 7f4bc6826700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bac02ff10 0x7f4bac0323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:21.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.818+0000 7f4bc6826700 1 --2- 172.21.15.118:0/380656438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bc00f51a0 0x7f4bc0109290 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:21.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.818+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 >> 172.21.15.118:0/380656438 conn(0x7f4bc00f09a0 msgr2=0x7f4bc00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:21.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.818+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 shutdown_connections 2024-02-23T00:25:21.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:21.818+0000 7f4bc6826700 1 -- 172.21.15.118:0/380656438 wait complete. 2024-02-23T00:25:21.820 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:22.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:21.816965+0000 mon.smithi090 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.118:0/380656438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:23.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:23.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:23.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:23.128411+0000 mgr.smithi090.pxxmlz (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 -- 172.21.15.118:0/2385027304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 msgr2=0x7fd5b80f8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 --2- 172.21.15.118:0/2385027304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b80f8e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fd5b4009fb0 tx=0x7fd5b4009b30).stop 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 -- 172.21.15.118:0/2385027304 shutdown_connections 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 --2- 172.21.15.118:0/2385027304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b80f8e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 -- 172.21.15.118:0/2385027304 >> 172.21.15.118:0/2385027304 conn(0x7fd5b80f08c0 msgr2=0x7fd5b80f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 -- 172.21.15.118:0/2385027304 shutdown_connections 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 -- 172.21.15.118:0/2385027304 wait complete. 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.498+0000 7fd5bed73700 1 Processor -- start 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bed73700 1 -- start start 2024-02-23T00:25:25.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bed73700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bed73700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b8106a10 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bdd71700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bdd71700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60752/0 (socket says 172.21.15.118:60752) 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bdd71700 1 -- 172.21.15.118:0/1990327024 learned_addr learned my addr 172.21.15.118:0/1990327024 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bdd71700 1 -- 172.21.15.118:0/1990327024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4005040 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5bdd71700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd5b40089f0 tx=0x7fd5b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.499+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b4009320 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.500+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b8106c10 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.500+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5b4013380 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.500+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b4014e20 con 0x7fd5b8060280 2024-02-23T00:25:25.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.500+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b81070a0 con 0x7fd5b8060280 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.500+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5b40263a0 con 0x7fd5b8060280 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.501+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b8045ad0 con 0x7fd5b8060280 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.501+0000 7fd5ae7fc700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 0x7fd5a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.501+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5b4041290 con 0x7fd5b8060280 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.502+0000 7fd5bd570700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 0x7fd5a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:25.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.502+0000 7fd5bd570700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 0x7fd5a40323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd5a80096a0 tx=0x7fd5a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:25.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.505+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5b4012b50 con 0x7fd5b8060280 2024-02-23T00:25:25.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:25.128732+0000 mgr.smithi090.pxxmlz (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:25.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.709+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5b8107ef0 con 0x7fd5b8060280 2024-02-23T00:25:25.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.710+0000 7fd5ae7fc700 1 -- 172.21.15.118:0/1990327024 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5b4012b50 con 0x7fd5b8060280 2024-02-23T00:25:25.711 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:25.712 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.711+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 msgr2=0x7fd5a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.711+0000 7fd5bed73700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 0x7fd5a40323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd5a80096a0 tx=0x7fd5a8006b40).stop 2024-02-23T00:25:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.711+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 msgr2=0x7fd5b8106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.711+0000 7fd5bed73700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd5b40089f0 tx=0x7fd5b400b040).stop 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 shutdown_connections 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd5a402ff10 0x7fd5a40323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 --2- 172.21.15.118:0/1990327024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5b8060280 0x7fd5b8106520 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 >> 172.21.15.118:0/1990327024 conn(0x7fd5b80f08c0 msgr2=0x7fd5b80f1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 shutdown_connections 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:25.712+0000 7fd5bed73700 1 -- 172.21.15.118:0/1990327024 wait complete. 2024-02-23T00:25:25.714 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:26.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:25.242804+0000 mon.smithi090 (mon.0) 138 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:25:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:25.250957+0000 mon.smithi090 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:25:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:25.711026+0000 mon.smithi090 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.118:0/1990327024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:27.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:27.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:27.129084+0000 mgr.smithi090.pxxmlz (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 -- 172.21.15.118:0/1763629531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 msgr2=0x7fb3bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/1763629531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc0f55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb3ac009fb0 tx=0x7fb3ac009b30).stop 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 -- 172.21.15.118:0/1763629531 shutdown_connections 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/1763629531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc0f55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 -- 172.21.15.118:0/1763629531 >> 172.21.15.118:0/1763629531 conn(0x7fb3bc0f09a0 msgr2=0x7fb3bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 -- 172.21.15.118:0/1763629531 shutdown_connections 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.362+0000 7fb3bb59e700 1 -- 172.21.15.118:0/1763629531 wait complete. 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3bb59e700 1 Processor -- start 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3bb59e700 1 -- start start 2024-02-23T00:25:29.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3bb59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3bb59e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc060280 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3ba59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3ba59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60780/0 (socket says 172.21.15.118:60780) 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3ba59c700 1 -- 172.21.15.118:0/3634126018 learned_addr learned my addr 172.21.15.118:0/3634126018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.363+0000 7fb3ba59c700 1 -- 172.21.15.118:0/3634126018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac005040 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3ba59c700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb3ac0089f0 tx=0x7fb3ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac009320 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc060480 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3ac013380 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac014e20 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.364+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc1099c0 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3ac0095f0 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc045ad0 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3aaffd700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 0x7fb3a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:29.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3ac040f70 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3b9d9b700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 0x7fb3a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:29.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.365+0000 7fb3b9d9b700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 0x7fb3a40323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:29.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.369+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ac06d2b0 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:29.129436+0000 mgr.smithi090.pxxmlz (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:29.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.573+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3bc0fad30 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.574+0000 7fb3aaffd700 1 -- 172.21.15.118:0/3634126018 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3ac014420 con 0x7fb3bc0f51a0 2024-02-23T00:25:29.575 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:29.576 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.575+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 msgr2=0x7fb3a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.575+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 0x7fb3a40323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).stop 2024-02-23T00:25:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.575+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 msgr2=0x7fb3bc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.575+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb3ac0089f0 tx=0x7fb3ac00b040).stop 2024-02-23T00:25:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 shutdown_connections 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3a402ff10 0x7fb3a40323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 --2- 172.21.15.118:0/3634126018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3bc0f51a0 0x7fb3bc109290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 >> 172.21.15.118:0/3634126018 conn(0x7fb3bc0f09a0 msgr2=0x7fb3bc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 shutdown_connections 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:29.576+0000 7fb3bb59e700 1 -- 172.21.15.118:0/3634126018 wait complete. 2024-02-23T00:25:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:30.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:29.574741+0000 mon.smithi090 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.118:0/3634126018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:31.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:31.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:31.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:31.129791+0000 mgr.smithi090.pxxmlz (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 -- 172.21.15.118:0/1194891839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 msgr2=0x7f577c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 --2- 172.21.15.118:0/1194891839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c0f6930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5770009fb0 tx=0x7f5770009b30).stop 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 -- 172.21.15.118:0/1194891839 shutdown_connections 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 --2- 172.21.15.118:0/1194891839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c0f6930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 -- 172.21.15.118:0/1194891839 >> 172.21.15.118:0/1194891839 conn(0x7f577c0f1c70 msgr2=0x7f577c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 -- 172.21.15.118:0/1194891839 shutdown_connections 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.339+0000 7f5781d56700 1 -- 172.21.15.118:0/1194891839 wait complete. 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.340+0000 7f5781d56700 1 Processor -- start 2024-02-23T00:25:33.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.340+0000 7f5781d56700 1 -- start start 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.340+0000 7f5781d56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.340+0000 7f5781d56700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c106d00 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.340+0000 7f577b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f577b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60802/0 (socket says 172.21.15.118:60802) 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f577b7fe700 1 -- 172.21.15.118:0/1421056378 learned_addr learned my addr 172.21.15.118:0/1421056378 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f577b7fe700 1 -- 172.21.15.118:0/1421056378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5770005040 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f577b7fe700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f57700089f0 tx=0x7f577000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5770009320 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c106f00 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5770013380 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f577001ed80 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.341+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c107390 con 0x7f577c0f6510 2024-02-23T00:25:33.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.342+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f577001e420 con 0x7f577c0f6510 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.342+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c045ad0 con 0x7f577c0f6510 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.342+0000 7f5763fff700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 0x7f5764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.342+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5770041090 con 0x7f577c0f6510 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.342+0000 7f577affd700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 0x7f5764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.343+0000 7f577affd700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 0x7f5764032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f576c0096a0 tx=0x7f576c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:33.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.347+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57700095f0 con 0x7f577c0f6510 2024-02-23T00:25:33.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:33.130141+0000 mgr.smithi090.pxxmlz (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:33.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.553+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f577c108240 con 0x7f577c0f6510 2024-02-23T00:25:33.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.554+0000 7f5763fff700 1 -- 172.21.15.118:0/1421056378 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5770004490 con 0x7f577c0f6510 2024-02-23T00:25:33.557 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:33.557 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:33.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 msgr2=0x7f5764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 0x7f5764032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f576c0096a0 tx=0x7f576c006b40).stop 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 msgr2=0x7f577c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f57700089f0 tx=0x7f577000b040).stop 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 shutdown_connections 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f576402ff60 0x7f5764032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 --2- 172.21.15.118:0/1421056378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f577c0f6510 0x7f577c106810 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 >> 172.21.15.118:0/1421056378 conn(0x7f577c0f1c70 msgr2=0x7f577c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 shutdown_connections 2024-02-23T00:25:33.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:33.556+0000 7f5781d56700 1 -- 172.21.15.118:0/1421056378 wait complete. 2024-02-23T00:25:33.571 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:34.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:33.555859+0000 mon.smithi090 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.118:0/1421056378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:35.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:35.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:35.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:35.130470+0000 mgr.smithi090.pxxmlz (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.185+0000 7fefa66df700 1 -- 172.21.15.118:0/3537305128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 msgr2=0x7fefa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.185+0000 7fefa66df700 1 --2- 172.21.15.118:0/3537305128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa00f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fef90009fb0 tx=0x7fef90009b30).stop 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- 172.21.15.118:0/3537305128 shutdown_connections 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 --2- 172.21.15.118:0/3537305128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa00f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- 172.21.15.118:0/3537305128 >> 172.21.15.118:0/3537305128 conn(0x7fefa00f0b70 msgr2=0x7fefa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- 172.21.15.118:0/3537305128 shutdown_connections 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- 172.21.15.118:0/3537305128 wait complete. 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 Processor -- start 2024-02-23T00:25:37.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- start start 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.186+0000 7fefa66df700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa0060240 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:46834/0 (socket says 172.21.15.118:46834) 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9ffff700 1 -- 172.21.15.118:0/3506679112 learned_addr learned my addr 172.21.15.118:0/3506679112 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9ffff700 1 -- 172.21.15.118:0/3506679112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90005040 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9ffff700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fef90005010 tx=0x7fef9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef90009320 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa0060440 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef90013380 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef90014e50 con 0x7fefa00f5410 2024-02-23T00:25:37.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.187+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa0109bc0 con 0x7fefa00f5410 2024-02-23T00:25:37.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.188+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef900263a0 con 0x7fefa00f5410 2024-02-23T00:25:37.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.188+0000 7fef9cff9700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 0x7fef880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.188+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef90040ef0 con 0x7fefa00f5410 2024-02-23T00:25:37.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.188+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa0045ad0 con 0x7fefa00f5410 2024-02-23T00:25:37.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.188+0000 7fef9f7fe700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 0x7fef880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.189+0000 7fef9f7fe700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 0x7fef880323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fef940096a0 tx=0x7fef94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:37.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.193+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef90012920 con 0x7fefa00f5410 2024-02-23T00:25:37.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.398+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefa00fb090 con 0x7fefa00f5410 2024-02-23T00:25:37.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.398+0000 7fef9cff9700 1 -- 172.21.15.118:0/3506679112 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef9002b020 con 0x7fefa00f5410 2024-02-23T00:25:37.401 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:37.401 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:37.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 msgr2=0x7fef880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 0x7fef880323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fef940096a0 tx=0x7fef94006b40).stop 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 msgr2=0x7fefa0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fef90005010 tx=0x7fef9000b040).stop 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 shutdown_connections 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fef8802ff10 0x7fef880323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 --2- 172.21.15.118:0/3506679112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefa00f5410 0x7fefa0109510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 >> 172.21.15.118:0/3506679112 conn(0x7fefa00f0b70 msgr2=0x7fefa00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 shutdown_connections 2024-02-23T00:25:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:37.400+0000 7fefa66df700 1 -- 172.21.15.118:0/3506679112 wait complete. 2024-02-23T00:25:37.404 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:37.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:37.130789+0000 mgr.smithi090.pxxmlz (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:38.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:37.400287+0000 mon.smithi090 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.118:0/3506679112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:39.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:39.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:39.131155+0000 mgr.smithi090.pxxmlz (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:41.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- 172.21.15.118:0/1200041017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 msgr2=0x7fe8140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 --2- 172.21.15.118:0/1200041017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe810009fb0 tx=0x7fe810009b30).stop 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- 172.21.15.118:0/1200041017 shutdown_connections 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 --2- 172.21.15.118:0/1200041017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- 172.21.15.118:0/1200041017 >> 172.21.15.118:0/1200041017 conn(0x7fe8140f09a0 msgr2=0x7fe8140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- 172.21.15.118:0/1200041017 shutdown_connections 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- 172.21.15.118:0/1200041017 wait complete. 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 Processor -- start 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- start start 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.195+0000 7fe81ad67700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8140fb6f0 con 0x7fe8140f51a0 2024-02-23T00:25:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe819d65700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe819d65700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:46856/0 (socket says 172.21.15.118:46856) 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe819d65700 1 -- 172.21.15.118:0/3787964475 learned_addr learned my addr 172.21.15.118:0/3787964475 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe819d65700 1 -- 172.21.15.118:0/3787964475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe810005040 con 0x7fe8140f51a0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe819d65700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe8100089f0 tx=0x7fe81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810009320 con 0x7fe8140f51a0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8140fb890 con 0x7fe8140f51a0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8140fbd20 con 0x7fe8140f51a0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe810013380 con 0x7fe8140f51a0 2024-02-23T00:25:41.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.196+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810014e40 con 0x7fe8140f51a0 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.197+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe810014420 con 0x7fe8140f51a0 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.197+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe814059100 con 0x7fe8140f51a0 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.197+0000 7fe80a7fc700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 0x7fe8000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.197+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe810040f50 con 0x7fe8140f51a0 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.197+0000 7fe819564700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 0x7fe8000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:41.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.198+0000 7fe819564700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 0x7fe8000323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe8040096a0 tx=0x7fe804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:41.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.202+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe81006d2b0 con 0x7fe8140f51a0 2024-02-23T00:25:41.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.409+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8140fad30 con 0x7fe8140f51a0 2024-02-23T00:25:41.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.409+0000 7fe80a7fc700 1 -- 172.21.15.118:0/3787964475 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe810009680 con 0x7fe8140f51a0 2024-02-23T00:25:41.411 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:41.411 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:41.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 msgr2=0x7fe8000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 0x7fe8000323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe8040096a0 tx=0x7fe804006b40).stop 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 msgr2=0x7fe8140fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe8100089f0 tx=0x7fe81000b040).stop 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 shutdown_connections 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe80002ff10 0x7fe8000323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 --2- 172.21.15.118:0/3787964475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8140f51a0 0x7fe8140fd270 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 >> 172.21.15.118:0/3787964475 conn(0x7fe8140f09a0 msgr2=0x7fe8140f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 shutdown_connections 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:41.411+0000 7fe81ad67700 1 -- 172.21.15.118:0/3787964475 wait complete. 2024-02-23T00:25:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:41.131537+0000 mgr.smithi090.pxxmlz (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:41.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:41.410286+0000 mon.smithi090 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.118:0/3787964475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:43.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:43.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:43.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:43.131891+0000 mgr.smithi090.pxxmlz (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- 172.21.15.118:0/2311983789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 msgr2=0x7f8a240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/2311983789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a240f90f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f8a0c009fb0 tx=0x7f8a0c009b30).stop 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- 172.21.15.118:0/2311983789 shutdown_connections 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/2311983789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a240f90f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- 172.21.15.118:0/2311983789 >> 172.21.15.118:0/2311983789 conn(0x7f8a240f0b70 msgr2=0x7f8a240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- 172.21.15.118:0/2311983789 shutdown_connections 2024-02-23T00:25:45.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- 172.21.15.118:0/2311983789 wait complete. 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 Processor -- start 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.204+0000 7f8a29d4a700 1 -- start start 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a29d4a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a29d4a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24106c70 con 0x7f8a24060280 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a237fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a237fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33224/0 (socket says 172.21.15.118:33224) 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a237fe700 1 -- 172.21.15.118:0/304330810 learned_addr learned my addr 172.21.15.118:0/304330810 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a237fe700 1 -- 172.21.15.118:0/304330810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c005040 con 0x7f8a24060280 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.205+0000 7f8a237fe700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f8a0c0089f0 tx=0x7f8a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a0c009320 con 0x7f8a24060280 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24106e70 con 0x7f8a24060280 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a0c013380 con 0x7f8a24060280 2024-02-23T00:25:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a0c014e20 con 0x7f8a24060280 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a24107300 con 0x7f8a24060280 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a0c0095f0 con 0x7f8a24060280 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.206+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24045ad0 con 0x7f8a24060280 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.207+0000 7f8a1bfff700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 0x7f8a100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.207+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a0c040f10 con 0x7f8a24060280 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.207+0000 7f8a22ffd700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 0x7f8a100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.207+0000 7f8a22ffd700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 0x7f8a100323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8a140096a0 tx=0x7f8a14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:45.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.211+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a0c014420 con 0x7f8a24060280 2024-02-23T00:25:45.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.421+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a241081b0 con 0x7f8a24060280 2024-02-23T00:25:45.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.422+0000 7f8a1bfff700 1 -- 172.21.15.118:0/304330810 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a0c01d980 con 0x7f8a24060280 2024-02-23T00:25:45.423 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:45.423 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:45.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 msgr2=0x7f8a100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 0x7f8a100323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8a140096a0 tx=0x7f8a14006b40).stop 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 msgr2=0x7f8a24106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f8a0c0089f0 tx=0x7f8a0c00b040).stop 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 shutdown_connections 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a1002ff10 0x7f8a100323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 --2- 172.21.15.118:0/304330810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a24060280 0x7f8a24106780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 >> 172.21.15.118:0/304330810 conn(0x7f8a240f0b70 msgr2=0x7f8a240f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 shutdown_connections 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:45.423+0000 7f8a29d4a700 1 -- 172.21.15.118:0/304330810 wait complete. 2024-02-23T00:25:45.428 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:45.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:45.132218+0000 mgr.smithi090.pxxmlz (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:46.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:45.422682+0000 mon.smithi090 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.118:0/304330810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:47.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:47.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:47.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:47.132528+0000 mgr.smithi090.pxxmlz (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 -- 172.21.15.118:0/3482183498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 msgr2=0x7fce380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/3482183498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce380f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fce2c009fb0 tx=0x7fce2c009b30).stop 2024-02-23T00:25:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 -- 172.21.15.118:0/3482183498 shutdown_connections 2024-02-23T00:25:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/3482183498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce380f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 -- 172.21.15.118:0/3482183498 >> 172.21.15.118:0/3482183498 conn(0x7fce380f0b70 msgr2=0x7fce380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 -- 172.21.15.118:0/3482183498 shutdown_connections 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 -- 172.21.15.118:0/3482183498 wait complete. 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.956+0000 7fce3ebc6700 1 Processor -- start 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3ebc6700 1 -- start start 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3ebc6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3ebc6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce38060220 con 0x7fce380f5410 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3c962700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3c962700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33244/0 (socket says 172.21.15.118:33244) 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3c962700 1 -- 172.21.15.118:0/2990295001 learned_addr learned my addr 172.21.15.118:0/2990295001 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.957+0000 7fce3c962700 1 -- 172.21.15.118:0/2990295001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c005040 con 0x7fce380f5410 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce3c962700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fce2c005010 tx=0x7fce2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce2c009320 con 0x7fce380f5410 2024-02-23T00:25:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce38060420 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce2c013380 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce2c014e50 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.958+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce38109bf0 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce2c014420 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce38059100 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce357fa700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 0x7fce280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce2c040e30 con 0x7fce380f5410 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce37fff700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 0x7fce280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.959+0000 7fce37fff700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 0x7fce280323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fce240096a0 tx=0x7fce24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:48.963+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce2c0095f0 con 0x7fce380f5410 2024-02-23T00:25:49.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.169+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce38045ad0 con 0x7fce380f5410 2024-02-23T00:25:49.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.170+0000 7fce357fa700 1 -- 172.21.15.118:0/2990295001 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce2c01db70 con 0x7fce380f5410 2024-02-23T00:25:49.171 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:49.171 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:49.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 msgr2=0x7fce280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 0x7fce280323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fce240096a0 tx=0x7fce24006b40).stop 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 msgr2=0x7fce38109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fce2c005010 tx=0x7fce2c00b040).stop 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 shutdown_connections 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fce2802ff10 0x7fce280323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 --2- 172.21.15.118:0/2990295001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce380f5410 0x7fce38109540 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 >> 172.21.15.118:0/2990295001 conn(0x7fce380f0b70 msgr2=0x7fce380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 shutdown_connections 2024-02-23T00:25:49.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:49.171+0000 7fce3ebc6700 1 -- 172.21.15.118:0/2990295001 wait complete. 2024-02-23T00:25:49.174 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:49.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:49.132876+0000 mgr.smithi090.pxxmlz (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:49.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:49.170792+0000 mon.smithi090 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.118:0/2990295001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:50.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:50.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:51.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:51.133114+0000 mgr.smithi090.pxxmlz (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 -- 172.21.15.118:0/1427300621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 msgr2=0x7f8ab40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 --2- 172.21.15.118:0/1427300621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab40f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8ab0009fb0 tx=0x7f8ab0009b30).stop 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 -- 172.21.15.118:0/1427300621 shutdown_connections 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 --2- 172.21.15.118:0/1427300621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab40f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 -- 172.21.15.118:0/1427300621 >> 172.21.15.118:0/1427300621 conn(0x7f8ab40f0b70 msgr2=0x7f8ab40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 -- 172.21.15.118:0/1427300621 shutdown_connections 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.009+0000 7f8abb827700 1 -- 172.21.15.118:0/1427300621 wait complete. 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8abb827700 1 Processor -- start 2024-02-23T00:25:53.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8abb827700 1 -- start start 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8abb827700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8abb827700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4060240 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8ab95c3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8ab95c3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33268/0 (socket says 172.21.15.118:33268) 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8ab95c3700 1 -- 172.21.15.118:0/1310480725 learned_addr learned my addr 172.21.15.118:0/1310480725 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8ab95c3700 1 -- 172.21.15.118:0/1310480725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab0005040 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8ab95c3700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8ab0005010 tx=0x7f8ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab0009320 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab4060440 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ab0013380 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.010+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab0014e50 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.011+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab4109b90 con 0x7f8ab40f5410 2024-02-23T00:25:53.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.011+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ab00263a0 con 0x7f8ab40f5410 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.012+0000 7f8aa9ffb700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 0x7f8aa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.012+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ab0040e80 con 0x7f8ab40f5410 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.012+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab4045ad0 con 0x7f8ab40f5410 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.012+0000 7f8ab8dc2700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 0x7f8aa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.012+0000 7f8ab8dc2700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 0x7f8aa00323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8aa40096a0 tx=0x7f8aa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:53.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.016+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ab0012920 con 0x7f8ab40f5410 2024-02-23T00:25:53.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.225+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ab40fb090 con 0x7f8ab40f5410 2024-02-23T00:25:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.225+0000 7f8aa9ffb700 1 -- 172.21.15.118:0/1310480725 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ab002b020 con 0x7f8ab40f5410 2024-02-23T00:25:53.227 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:53.227 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 msgr2=0x7f8aa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 0x7f8aa00323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8aa40096a0 tx=0x7f8aa4006b40).stop 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 msgr2=0x7f8ab41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8ab0005010 tx=0x7f8ab000b040).stop 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 shutdown_connections 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8aa002ff10 0x7f8aa00323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 --2- 172.21.15.118:0/1310480725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f5410 0x7f8ab41094e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 >> 172.21.15.118:0/1310480725 conn(0x7f8ab40f0b70 msgr2=0x7f8ab40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 shutdown_connections 2024-02-23T00:25:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:53.227+0000 7f8abb827700 1 -- 172.21.15.118:0/1310480725 wait complete. 2024-02-23T00:25:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:53.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:53.133398+0000 mgr.smithi090.pxxmlz (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:54.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:53.226281+0000 mon.smithi090 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.118:0/1310480725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:54.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:54.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:55.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:55.133785+0000 mgr.smithi090.pxxmlz (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:56.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 -- 172.21.15.118:0/3751424077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 msgr2=0x7fd8a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:56.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 --2- 172.21.15.118:0/3751424077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a40f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd894009fb0 tx=0x7fd894009b30).stop 2024-02-23T00:25:56.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 -- 172.21.15.118:0/3751424077 shutdown_connections 2024-02-23T00:25:56.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 --2- 172.21.15.118:0/3751424077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a40f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:56.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 -- 172.21.15.118:0/3751424077 >> 172.21.15.118:0/3751424077 conn(0x7fd8a40f0b70 msgr2=0x7fd8a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:56.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 -- 172.21.15.118:0/3751424077 shutdown_connections 2024-02-23T00:25:56.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.941+0000 7fd8aa532700 1 -- 172.21.15.118:0/3751424077 wait complete. 2024-02-23T00:25:56.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.942+0000 7fd8aa532700 1 Processor -- start 2024-02-23T00:25:56.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.942+0000 7fd8aa532700 1 -- start start 2024-02-23T00:25:56.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.942+0000 7fd8aa532700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:56.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.942+0000 7fd8aa532700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4105bd0 con 0x7fd8a40f5410 2024-02-23T00:25:56.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:43496/0 (socket says 172.21.15.118:43496) 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a3fff700 1 -- 172.21.15.118:0/2367714301 learned_addr learned my addr 172.21.15.118:0/2367714301 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a3fff700 1 -- 172.21.15.118:0/2367714301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894005040 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a3fff700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd894005010 tx=0x7fd89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd894009320 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a4105dd0 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd894013380 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd894014e50 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.943+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a4106260 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.944+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd894014420 con 0x7fd8a40f5410 2024-02-23T00:25:56.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.944+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a4045ad0 con 0x7fd8a40f5410 2024-02-23T00:25:56.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.944+0000 7fd8a0ff9700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 0x7fd8900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:56.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.944+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd894040f90 con 0x7fd8a40f5410 2024-02-23T00:25:56.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.944+0000 7fd8a37fe700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 0x7fd8900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:56.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.945+0000 7fd8a37fe700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 0x7fd8900323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd88c0096a0 tx=0x7fd88c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:56.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:56.949+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd894014790 con 0x7fd8a40f5410 2024-02-23T00:25:57.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.153+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8a4107110 con 0x7fd8a40f5410 2024-02-23T00:25:57.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.153+0000 7fd8a0ff9700 1 -- 172.21.15.118:0/2367714301 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd89401d380 con 0x7fd8a40f5410 2024-02-23T00:25:57.155 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:25:57.155 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:25:57.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 msgr2=0x7fd8900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:57.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 0x7fd8900323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd88c0096a0 tx=0x7fd88c006b40).stop 2024-02-23T00:25:57.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 msgr2=0x7fd8a41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd894005010 tx=0x7fd89400b040).stop 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 shutdown_connections 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd89002ff10 0x7fd8900323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 --2- 172.21.15.118:0/2367714301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8a40f5410 0x7fd8a41056e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 >> 172.21.15.118:0/2367714301 conn(0x7fd8a40f0b70 msgr2=0x7fd8a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 shutdown_connections 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:25:57.155+0000 7fd8aa532700 1 -- 172.21.15.118:0/2367714301 wait complete. 2024-02-23T00:25:57.157 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:25:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:57.134129+0000 mgr.smithi090.pxxmlz (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:25:57.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:25:57.154297+0000 mon.smithi090 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.118:0/2367714301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:25:58.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:25:58.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:25:59.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:25:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:25:59.134421+0000 mgr.smithi090.pxxmlz (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- 172.21.15.118:0/2605290537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 msgr2=0x7f37ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2605290537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f37dc009fb0 tx=0x7f37dc009b30).stop 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- 172.21.15.118:0/2605290537 shutdown_connections 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2605290537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- 172.21.15.118:0/2605290537 >> 172.21.15.118:0/2605290537 conn(0x7f37ec0f0b70 msgr2=0x7f37ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- 172.21.15.118:0/2605290537 shutdown_connections 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- 172.21.15.118:0/2605290537 wait complete. 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 Processor -- start 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- start start 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:00.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.835+0000 7f37f40c0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec106cc0 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f1e5c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f1e5c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:43508/0 (socket says 172.21.15.118:43508) 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f1e5c700 1 -- 172.21.15.118:0/2812983666 learned_addr learned my addr 172.21.15.118:0/2812983666 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f1e5c700 1 -- 172.21.15.118:0/2812983666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc005040 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f1e5c700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f37dc005010 tx=0x7f37dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc009320 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec106ec0 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37dc013380 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37dc014e50 con 0x7f37ec0f5410 2024-02-23T00:26:00.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.836+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec107350 con 0x7f37ec0f5410 2024-02-23T00:26:00.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.837+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37dc0095f0 con 0x7f37ec0f5410 2024-02-23T00:26:00.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.837+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec045ad0 con 0x7f37ec0f5410 2024-02-23T00:26:00.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.837+0000 7f37e27fc700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 0x7f37d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:00.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.837+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37dc040fa0 con 0x7f37ec0f5410 2024-02-23T00:26:00.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.837+0000 7f37f165b700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 0x7f37d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:00.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.838+0000 7f37f165b700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 0x7f37d80323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f37e80096a0 tx=0x7f37e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:00.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:00.842+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37dc014420 con 0x7f37ec0f5410 2024-02-23T00:26:01.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.049+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37ec108200 con 0x7f37ec0f5410 2024-02-23T00:26:01.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.049+0000 7f37e27fc700 1 -- 172.21.15.118:0/2812983666 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37dc010230 con 0x7f37ec0f5410 2024-02-23T00:26:01.051 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:01.051 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 msgr2=0x7f37d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 0x7f37d80323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f37e80096a0 tx=0x7f37e8006b40).stop 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 msgr2=0x7f37ec1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f37dc005010 tx=0x7f37dc00b040).stop 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 shutdown_connections 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f37d802ff10 0x7f37d80323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 --2- 172.21.15.118:0/2812983666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37ec0f5410 0x7f37ec1067d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 >> 172.21.15.118:0/2812983666 conn(0x7f37ec0f0b70 msgr2=0x7f37ec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 shutdown_connections 2024-02-23T00:26:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:01.051+0000 7f37f40c0700 1 -- 172.21.15.118:0/2812983666 wait complete. 2024-02-23T00:26:01.054 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:01.050245+0000 mon.smithi090 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.118:0/2812983666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:01.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:01.134725+0000 mgr.smithi090.pxxmlz (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:02.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:02.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:03.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:03.135003+0000 mgr.smithi090.pxxmlz (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- 172.21.15.118:0/3358997183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 msgr2=0x7fb3f80f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 --2- 172.21.15.118:0/3358997183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80f5df0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb3e8009fb0 tx=0x7fb3e8009b30).stop 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- 172.21.15.118:0/3358997183 shutdown_connections 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 --2- 172.21.15.118:0/3358997183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80f5df0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- 172.21.15.118:0/3358997183 >> 172.21.15.118:0/3358997183 conn(0x7fb3f80f02e0 msgr2=0x7fb3f80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- 172.21.15.118:0/3358997183 shutdown_connections 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- 172.21.15.118:0/3358997183 wait complete. 2024-02-23T00:26:04.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 Processor -- start 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- start start 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.893+0000 7fb400767700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80fb950 con 0x7fb3f80f59d0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3fe503700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3fe503700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:51884/0 (socket says 172.21.15.118:51884) 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3fe503700 1 -- 172.21.15.118:0/813561631 learned_addr learned my addr 172.21.15.118:0/813561631 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3fe503700 1 -- 172.21.15.118:0/813561631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e8005040 con 0x7fb3f80f59d0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3fe503700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb3e8005010 tx=0x7fb3e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8009320 con 0x7fb3f80f59d0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f80fbaf0 con 0x7fb3f80f59d0 2024-02-23T00:26:04.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3e8013380 con 0x7fb3f80f59d0 2024-02-23T00:26:04.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8014e50 con 0x7fb3f80f59d0 2024-02-23T00:26:04.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.894+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f80fbf80 con 0x7fb3f80f59d0 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.895+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3e80263a0 con 0x7fb3f80f59d0 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.895+0000 7fb3eeffd700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.895+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8059100 con 0x7fb3f80f59d0 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.895+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3e8040f90 con 0x7fb3f80f59d0 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.895+0000 7fb3fdd02700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:04.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.896+0000 7fb3fdd02700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 0x7fb3e40323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb3f40096a0 tx=0x7fb3f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:04.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:04.900+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3e8012920 con 0x7fb3f80f59d0 2024-02-23T00:26:05.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.107+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3f80faf90 con 0x7fb3f80f59d0 2024-02-23T00:26:05.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.108+0000 7fb3eeffd700 1 -- 172.21.15.118:0/813561631 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3e802b020 con 0x7fb3f80f59d0 2024-02-23T00:26:05.110 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:05.110 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:05.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 msgr2=0x7fb3e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:05.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 0x7fb3e40323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb3f40096a0 tx=0x7fb3f4006b40).stop 2024-02-23T00:26:05.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 msgr2=0x7fb3f80fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb3e8005010 tx=0x7fb3e800b040).stop 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 shutdown_connections 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 --2- 172.21.15.118:0/813561631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3f80f59d0 0x7fb3f80fd4d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 >> 172.21.15.118:0/813561631 conn(0x7fb3f80f02e0 msgr2=0x7fb3f80f4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 shutdown_connections 2024-02-23T00:26:05.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:05.110+0000 7fb400767700 1 -- 172.21.15.118:0/813561631 wait complete. 2024-02-23T00:26:05.113 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:05.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:05.109196+0000 mon.smithi090 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.118:0/813561631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:06.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:05.135299+0000 mgr.smithi090.pxxmlz (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:06.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:06.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:08.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:07.135653+0000 mgr.smithi090.pxxmlz (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:08.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.731+0000 7ff113a10700 1 -- 172.21.15.118:0/3121563875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 msgr2=0x7ff10c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.731+0000 7ff113a10700 1 --2- 172.21.15.118:0/3121563875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c0f4b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff0fc009fb0 tx=0x7ff0fc009b30).stop 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.731+0000 7ff113a10700 1 -- 172.21.15.118:0/3121563875 shutdown_connections 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.731+0000 7ff113a10700 1 --2- 172.21.15.118:0/3121563875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c0f4b60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.731+0000 7ff113a10700 1 -- 172.21.15.118:0/3121563875 >> 172.21.15.118:0/3121563875 conn(0x7ff10c0f0260 msgr2=0x7ff10c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 -- 172.21.15.118:0/3121563875 shutdown_connections 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 -- 172.21.15.118:0/3121563875 wait complete. 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 Processor -- start 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 -- start start 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:08.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff113a10700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c106ca0 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff1117ac700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff1117ac700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:51900/0 (socket says 172.21.15.118:51900) 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.732+0000 7ff1117ac700 1 -- 172.21.15.118:0/2550505331 learned_addr learned my addr 172.21.15.118:0/2550505331 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff1117ac700 1 -- 172.21.15.118:0/2550505331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc005040 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff1117ac700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff0fc005010 tx=0x7ff0fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0fc009320 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c106ea0 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0fc013380 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0fc014e50 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.733+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c107330 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.734+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0fc0095f0 con 0x7ff10c0f4740 2024-02-23T00:26:08.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.734+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c045ad0 con 0x7ff10c0f4740 2024-02-23T00:26:08.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.734+0000 7ff101ffb700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 0x7ff0f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:08.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.734+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0fc040e90 con 0x7ff10c0f4740 2024-02-23T00:26:08.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.734+0000 7ff110fab700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 0x7ff0f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:08.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.735+0000 7ff110fab700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 0x7ff0f80323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff1080096a0 tx=0x7ff108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:08.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.739+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0fc014420 con 0x7ff10c0f4740 2024-02-23T00:26:08.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.948+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff10c108180 con 0x7ff10c0f4740 2024-02-23T00:26:08.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.948+0000 7ff101ffb700 1 -- 172.21.15.118:0/2550505331 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0fc010230 con 0x7ff10c0f4740 2024-02-23T00:26:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:08.950 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:08.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 msgr2=0x7ff0f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:08.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 0x7ff0f80323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff1080096a0 tx=0x7ff108006b40).stop 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 msgr2=0x7ff10c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff0fc005010 tx=0x7ff0fc00b040).stop 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 shutdown_connections 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff0f802ff10 0x7ff0f80323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 --2- 172.21.15.118:0/2550505331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10c0f4740 0x7ff10c1067b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 >> 172.21.15.118:0/2550505331 conn(0x7ff10c0f0260 msgr2=0x7ff10c0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 shutdown_connections 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:08.950+0000 7ff113a10700 1 -- 172.21.15.118:0/2550505331 wait complete. 2024-02-23T00:26:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:08.949262+0000 mon.smithi090 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.118:0/2550505331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:10.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:09.135922+0000 mgr.smithi090.pxxmlz (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:10.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:10.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:11.136264+0000 mgr.smithi090.pxxmlz (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:12.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.685+0000 7fd7b85a2700 1 -- 172.21.15.118:0/3538563983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 msgr2=0x7fd7b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.685+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/3538563983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b00f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd7a0009fb0 tx=0x7fd7a0009b30).stop 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.685+0000 7fd7b85a2700 1 -- 172.21.15.118:0/3538563983 shutdown_connections 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.685+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/3538563983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b00f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.685+0000 7fd7b85a2700 1 -- 172.21.15.118:0/3538563983 >> 172.21.15.118:0/3538563983 conn(0x7fd7b00f0b70 msgr2=0x7fd7b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 -- 172.21.15.118:0/3538563983 shutdown_connections 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 -- 172.21.15.118:0/3538563983 wait complete. 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 Processor -- start 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 -- start start 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b85a2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b0106cd0 con 0x7fd7b00f5410 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b633e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:12.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b633e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:51920/0 (socket says 172.21.15.118:51920) 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.686+0000 7fd7b633e700 1 -- 172.21.15.118:0/843466994 learned_addr learned my addr 172.21.15.118:0/843466994 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7b633e700 1 -- 172.21.15.118:0/843466994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0005040 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7b633e700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd7a0005010 tx=0x7fd7a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a0009320 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b0106ed0 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7a0013380 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a0014e50 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.687+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b0107360 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.688+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b0045ad0 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.688+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7a00095f0 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.688+0000 7fd7a6ffd700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 0x7fd79c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.688+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7a0040f70 con 0x7fd7b00f5410 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.688+0000 7fd7b5b3d700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 0x7fd79c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:12.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.689+0000 7fd7b5b3d700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 0x7fd79c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd7ac0096a0 tx=0x7fd7ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:12.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.693+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7a0014420 con 0x7fd7b00f5410 2024-02-23T00:26:12.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.897+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7b01081b0 con 0x7fd7b00f5410 2024-02-23T00:26:12.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.897+0000 7fd7a6ffd700 1 -- 172.21.15.118:0/843466994 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd7a0010230 con 0x7fd7b00f5410 2024-02-23T00:26:12.899 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:12.899 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:12.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 msgr2=0x7fd79c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 0x7fd79c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd7ac0096a0 tx=0x7fd7ac006b40).stop 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 msgr2=0x7fd7b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd7a0005010 tx=0x7fd7a000b040).stop 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 shutdown_connections 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd79c02ff10 0x7fd79c0323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 --2- 172.21.15.118:0/843466994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7b00f5410 0x7fd7b01067e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 >> 172.21.15.118:0/843466994 conn(0x7fd7b00f0b70 msgr2=0x7fd7b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 shutdown_connections 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:12.899+0000 7fd7b85a2700 1 -- 172.21.15.118:0/843466994 wait complete. 2024-02-23T00:26:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:13.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:12.898270+0000 mon.smithi090 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.118:0/843466994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:14.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:13.136610+0000 mgr.smithi090.pxxmlz (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:14.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:14.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:15.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:15.136992+0000 mgr.smithi090.pxxmlz (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:16.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 -- 172.21.15.118:0/73157144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 msgr2=0x7febe40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:16.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 --2- 172.21.15.118:0/73157144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe40f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7febd4009fb0 tx=0x7febd4009b30).stop 2024-02-23T00:26:16.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 -- 172.21.15.118:0/73157144 shutdown_connections 2024-02-23T00:26:16.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 --2- 172.21.15.118:0/73157144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe40f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:16.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 -- 172.21.15.118:0/73157144 >> 172.21.15.118:0/73157144 conn(0x7febe40f1c70 msgr2=0x7febe40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:16.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 -- 172.21.15.118:0/73157144 shutdown_connections 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.753+0000 7febec6c8700 1 -- 172.21.15.118:0/73157144 wait complete. 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 Processor -- start 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 -- start start 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4106cd0 con 0x7febe40f6510 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febea464700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febea464700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33666/0 (socket says 172.21.15.118:33666) 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febea464700 1 -- 172.21.15.118:0/3928513195 learned_addr learned my addr 172.21.15.118:0/3928513195 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febea464700 1 -- 172.21.15.118:0/3928513195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4005040 con 0x7febe40f6510 2024-02-23T00:26:16.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febea464700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7febd4005010 tx=0x7febd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd4009320 con 0x7febe40f6510 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe4106ed0 con 0x7febe40f6510 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febd4013380 con 0x7febe40f6510 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febd4014e50 con 0x7febe40f6510 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.754+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe4107360 con 0x7febe40f6510 2024-02-23T00:26:16.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.755+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febd4014420 con 0x7febe40f6510 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.755+0000 7febdaffd700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 0x7febd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.755+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febd4040e30 con 0x7febe40f6510 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.756+0000 7febe9c63700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 0x7febd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.756+0000 7febe9c63700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 0x7febd00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7febe00096a0 tx=0x7febe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.758+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe4059100 con 0x7febe40f6510 2024-02-23T00:26:16.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.762+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febd40095f0 con 0x7febe40f6510 2024-02-23T00:26:16.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.969+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febe40fc0a0 con 0x7febe40f6510 2024-02-23T00:26:16.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.969+0000 7febdaffd700 1 -- 172.21.15.118:0/3928513195 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febd402b020 con 0x7febe40f6510 2024-02-23T00:26:16.972 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:16.972 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:16.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 msgr2=0x7febd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:16.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 0x7febd00323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7febe00096a0 tx=0x7febe0006b40).stop 2024-02-23T00:26:16.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 msgr2=0x7febe41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:16.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7febd4005010 tx=0x7febd400b040).stop 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 shutdown_connections 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7febd002ff10 0x7febd00323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 --2- 172.21.15.118:0/3928513195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febe40f6510 0x7febe41067e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 >> 172.21.15.118:0/3928513195 conn(0x7febe40f1c70 msgr2=0x7febe40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 shutdown_connections 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:16.971+0000 7febec6c8700 1 -- 172.21.15.118:0/3928513195 wait complete. 2024-02-23T00:26:16.974 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:17.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:16.971421+0000 mon.smithi090 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.118:0/3928513195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:18.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:17.137363+0000 mgr.smithi090.pxxmlz (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:18.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:18.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:19.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:19.137742+0000 mgr.smithi090.pxxmlz (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 -- 172.21.15.118:0/2548317290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 msgr2=0x7fa7840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 --2- 172.21.15.118:0/2548317290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa7840f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa778009fb0 tx=0x7fa778009b30).stop 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 -- 172.21.15.118:0/2548317290 shutdown_connections 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 --2- 172.21.15.118:0/2548317290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa7840f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 -- 172.21.15.118:0/2548317290 >> 172.21.15.118:0/2548317290 conn(0x7fa7840f1c70 msgr2=0x7fa7840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 -- 172.21.15.118:0/2548317290 shutdown_connections 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.800+0000 7fa78beeb700 1 -- 172.21.15.118:0/2548317290 wait complete. 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa78beeb700 1 Processor -- start 2024-02-23T00:26:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa78beeb700 1 -- start start 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa78beeb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa78beeb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa784069e20 con 0x7fa7840f6510 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa789c87700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa789c87700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33680/0 (socket says 172.21.15.118:33680) 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa789c87700 1 -- 172.21.15.118:0/1240024075 learned_addr learned my addr 172.21.15.118:0/1240024075 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa789c87700 1 -- 172.21.15.118:0/1240024075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa778005040 con 0x7fa7840f6510 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa789c87700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa778005010 tx=0x7fa77800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa778009320 con 0x7fa7840f6510 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa784069fc0 con 0x7fa7840f6510 2024-02-23T00:26:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa778013380 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.801+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa778014e50 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.802+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78406a450 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.802+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa778014420 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.802+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa784059100 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.802+0000 7fa7767fc700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 0x7fa7700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.802+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa778040e30 con 0x7fa7840f6510 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.803+0000 7fa789486700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 0x7fa7700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.803+0000 7fa789486700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 0x7fa7700323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa7800096a0 tx=0x7fa780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:20.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:20.807+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7780095f0 con 0x7fa7840f6510 2024-02-23T00:26:21.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.016+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7840fc0a0 con 0x7fa7840f6510 2024-02-23T00:26:21.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.016+0000 7fa7767fc700 1 -- 172.21.15.118:0/1240024075 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa77801db70 con 0x7fa7840f6510 2024-02-23T00:26:21.019 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:21.019 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 msgr2=0x7fa7700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 0x7fa7700323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa7800096a0 tx=0x7fa780006b40).stop 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 msgr2=0x7fa78406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa778005010 tx=0x7fa77800b040).stop 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 shutdown_connections 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa77002ff10 0x7fa7700323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 --2- 172.21.15.118:0/1240024075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7840f6510 0x7fa78406b9a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 >> 172.21.15.118:0/1240024075 conn(0x7fa7840f1c70 msgr2=0x7fa7840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:21.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 shutdown_connections 2024-02-23T00:26:21.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:21.018+0000 7fa78beeb700 1 -- 172.21.15.118:0/1240024075 wait complete. 2024-02-23T00:26:21.022 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:21.018434+0000 mon.smithi090 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.118:0/1240024075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:21.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:21.138103+0000 mgr.smithi090.pxxmlz (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:22.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:22.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:23.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:23.138397+0000 mgr.smithi090.pxxmlz (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:24.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 -- 172.21.15.118:0/3499850229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 msgr2=0x7fdab40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:24.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 --2- 172.21.15.118:0/3499850229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab40f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fdaa8009fb0 tx=0x7fdaa8009b30).stop 2024-02-23T00:26:24.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 -- 172.21.15.118:0/3499850229 shutdown_connections 2024-02-23T00:26:24.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 --2- 172.21.15.118:0/3499850229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab40f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 -- 172.21.15.118:0/3499850229 >> 172.21.15.118:0/3499850229 conn(0x7fdab40f1c70 msgr2=0x7fdab40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 -- 172.21.15.118:0/3499850229 shutdown_connections 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.575+0000 7fdab9522700 1 -- 172.21.15.118:0/3499850229 wait complete. 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab9522700 1 Processor -- start 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab9522700 1 -- start start 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab9522700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab9522700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab4069e20 con 0x7fdab40f6510 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:34882/0 (socket says 172.21.15.118:34882) 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab2ffd700 1 -- 172.21.15.118:0/3401240425 learned_addr learned my addr 172.21.15.118:0/3401240425 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.576+0000 7fdab2ffd700 1 -- 172.21.15.118:0/3401240425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8005040 con 0x7fdab40f6510 2024-02-23T00:26:24.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fdab2ffd700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fdaa8005010 tx=0x7fdaa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:24.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa8009320 con 0x7fdab40f6510 2024-02-23T00:26:24.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab4069fc0 con 0x7fdab40f6510 2024-02-23T00:26:24.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa8013380 con 0x7fdab40f6510 2024-02-23T00:26:24.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaa8014e50 con 0x7fdab40f6510 2024-02-23T00:26:24.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab406a450 con 0x7fdab40f6510 2024-02-23T00:26:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.577+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaa8014420 con 0x7fdab40f6510 2024-02-23T00:26:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.578+0000 7fda9b7fe700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 0x7fda9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.578+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaa8040e30 con 0x7fdab40f6510 2024-02-23T00:26:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.578+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab4059100 con 0x7fdab40f6510 2024-02-23T00:26:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.578+0000 7fdab27fc700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 0x7fda9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:24.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.578+0000 7fdab27fc700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 0x7fda9c0323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fdaa40096a0 tx=0x7fdaa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:24.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.583+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaa80095f0 con 0x7fdab40f6510 2024-02-23T00:26:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.788+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdab406ab50 con 0x7fdab40f6510 2024-02-23T00:26:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.789+0000 7fda9b7fe700 1 -- 172.21.15.118:0/3401240425 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaa802b020 con 0x7fdab40f6510 2024-02-23T00:26:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:24.792 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 msgr2=0x7fda9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 0x7fda9c0323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fdaa40096a0 tx=0x7fdaa4006b40).stop 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 msgr2=0x7fdab406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fdaa8005010 tx=0x7fdaa800b040).stop 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 shutdown_connections 2024-02-23T00:26:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fda9c02ff10 0x7fda9c0323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:24.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 --2- 172.21.15.118:0/3401240425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdab40f6510 0x7fdab406b9a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:24.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 >> 172.21.15.118:0/3401240425 conn(0x7fdab40f1c70 msgr2=0x7fdab40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:24.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 shutdown_connections 2024-02-23T00:26:24.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:24.791+0000 7fdab9522700 1 -- 172.21.15.118:0/3401240425 wait complete. 2024-02-23T00:26:24.794 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:25.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:24.791009+0000 mon.smithi090 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.118:0/3401240425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:26.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:25.138753+0000 mgr.smithi090.pxxmlz (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:25.243872+0000 mon.smithi090 (mon.0) 156 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:26:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:25.251998+0000 mon.smithi090 (mon.0) 157 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:26:26.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:26.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:27.139130+0000 mgr.smithi090.pxxmlz (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 -- 172.21.15.118:0/2166203023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 msgr2=0x7f17cc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/2166203023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc0f4be0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f17c0009fb0 tx=0x7f17c0009b30).stop 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 -- 172.21.15.118:0/2166203023 shutdown_connections 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/2166203023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc0f4be0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 -- 172.21.15.118:0/2166203023 >> 172.21.15.118:0/2166203023 conn(0x7f17cc0f02e0 msgr2=0x7f17cc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 -- 172.21.15.118:0/2166203023 shutdown_connections 2024-02-23T00:26:28.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.509+0000 7f17d2ba8700 1 -- 172.21.15.118:0/2166203023 wait complete. 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.510+0000 7f17d2ba8700 1 Processor -- start 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.510+0000 7f17d2ba8700 1 -- start start 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.510+0000 7f17d2ba8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.510+0000 7f17d2ba8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc106cc0 con 0x7f17cc0f47c0 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d0944700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d0944700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:34900/0 (socket says 172.21.15.118:34900) 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d0944700 1 -- 172.21.15.118:0/1192487926 learned_addr learned my addr 172.21.15.118:0/1192487926 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:28.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d0944700 1 -- 172.21.15.118:0/1192487926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c0005040 con 0x7f17cc0f47c0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d0944700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f17c0005010 tx=0x7f17c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17c0009320 con 0x7f17cc0f47c0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc106ec0 con 0x7f17cc0f47c0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17c0013380 con 0x7f17cc0f47c0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17c0014e50 con 0x7f17cc0f47c0 2024-02-23T00:26:28.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.511+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc107350 con 0x7f17cc0f47c0 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.512+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17c00095f0 con 0x7f17cc0f47c0 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.512+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc045ad0 con 0x7f17cc0f47c0 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.512+0000 7f17c97fa700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 0x7f17bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.512+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17c0040e90 con 0x7f17cc0f47c0 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.513+0000 7f17cbfff700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 0x7f17bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:28.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.513+0000 7f17cbfff700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 0x7f17bc0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:28.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.517+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17c0014420 con 0x7f17cc0f47c0 2024-02-23T00:26:28.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.721+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17cc1081a0 con 0x7f17cc0f47c0 2024-02-23T00:26:28.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.721+0000 7f17c97fa700 1 -- 172.21.15.118:0/1192487926 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17c0010230 con 0x7f17cc0f47c0 2024-02-23T00:26:28.723 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:28.723 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:28.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 msgr2=0x7f17bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:28.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 0x7f17bc0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).stop 2024-02-23T00:26:28.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 msgr2=0x7f17cc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f17c0005010 tx=0x7f17c000b040).stop 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 shutdown_connections 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17bc02ff10 0x7f17bc0323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 --2- 172.21.15.118:0/1192487926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17cc0f47c0 0x7f17cc1067d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 >> 172.21.15.118:0/1192487926 conn(0x7f17cc0f02e0 msgr2=0x7f17cc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 shutdown_connections 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:28.723+0000 7f17d2ba8700 1 -- 172.21.15.118:0/1192487926 wait complete. 2024-02-23T00:26:28.725 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:29.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:28.722140+0000 mon.smithi090 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.118:0/1192487926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:30.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:29.139546+0000 mgr.smithi090.pxxmlz (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:30.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:30.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:31.139885+0000 mgr.smithi090.pxxmlz (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 -- 172.21.15.118:0/3526558567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 msgr2=0x7feb340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 --2- 172.21.15.118:0/3526558567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb340f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feb24009fb0 tx=0x7feb24009b30).stop 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 -- 172.21.15.118:0/3526558567 shutdown_connections 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 --2- 172.21.15.118:0/3526558567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb340f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 -- 172.21.15.118:0/3526558567 >> 172.21.15.118:0/3526558567 conn(0x7feb340f0b70 msgr2=0x7feb340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 -- 172.21.15.118:0/3526558567 shutdown_connections 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 -- 172.21.15.118:0/3526558567 wait complete. 2024-02-23T00:26:32.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.411+0000 7feb3c031700 1 Processor -- start 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb3c031700 1 -- start start 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb3c031700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb3c031700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb34106ca0 con 0x7feb340f5410 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb39dcd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb39dcd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:34912/0 (socket says 172.21.15.118:34912) 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb39dcd700 1 -- 172.21.15.118:0/3654236453 learned_addr learned my addr 172.21.15.118:0/3654236453 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb39dcd700 1 -- 172.21.15.118:0/3654236453 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24005040 con 0x7feb340f5410 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.412+0000 7feb39dcd700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7feb24005010 tx=0x7feb2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:32.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb24009320 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb34106ea0 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb24013380 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb24014e50 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb34107330 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.413+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb240095f0 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.414+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb34045ad0 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.414+0000 7feb2a7fc700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 0x7feb200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.414+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb24040e90 con 0x7feb340f5410 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.414+0000 7feb395cc700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 0x7feb200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.414+0000 7feb395cc700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 0x7feb200323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7feb300096a0 tx=0x7feb30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.418+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb24014420 con 0x7feb340f5410 2024-02-23T00:26:32.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.623+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb340fafa0 con 0x7feb340f5410 2024-02-23T00:26:32.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.624+0000 7feb2a7fc700 1 -- 172.21.15.118:0/3654236453 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb24010230 con 0x7feb340f5410 2024-02-23T00:26:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:32.626 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:32.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.625+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 msgr2=0x7feb200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.625+0000 7feb3c031700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 0x7feb200323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7feb300096a0 tx=0x7feb30006b40).stop 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 msgr2=0x7feb341067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7feb24005010 tx=0x7feb2400b040).stop 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 shutdown_connections 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb2002ff10 0x7feb200323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 --2- 172.21.15.118:0/3654236453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb340f5410 0x7feb341067b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 >> 172.21.15.118:0/3654236453 conn(0x7feb340f0b70 msgr2=0x7feb3405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 shutdown_connections 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:32.626+0000 7feb3c031700 1 -- 172.21.15.118:0/3654236453 wait complete. 2024-02-23T00:26:32.628 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:33.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:32.624943+0000 mon.smithi090 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.118:0/3654236453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:34.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:33.140264+0000 mgr.smithi090.pxxmlz (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:34.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:34.289 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:35.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:35.140686+0000 mgr.smithi090.pxxmlz (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:36.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 -- 172.21.15.118:0/437524480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 msgr2=0x7f27ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 --2- 172.21.15.118:0/437524480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec0f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f27e0009fb0 tx=0x7f27e0009b30).stop 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 -- 172.21.15.118:0/437524480 shutdown_connections 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 --2- 172.21.15.118:0/437524480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec0f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 -- 172.21.15.118:0/437524480 >> 172.21.15.118:0/437524480 conn(0x7f27ec0f0b70 msgr2=0x7f27ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 -- 172.21.15.118:0/437524480 shutdown_connections 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.451+0000 7f27f449e700 1 -- 172.21.15.118:0/437524480 wait complete. 2024-02-23T00:26:36.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f449e700 1 Processor -- start 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f449e700 1 -- start start 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f449e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f449e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec060240 con 0x7f27ec0f5410 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f223a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f223a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33730/0 (socket says 172.21.15.118:33730) 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f223a700 1 -- 172.21.15.118:0/2458831078 learned_addr learned my addr 172.21.15.118:0/2458831078 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f223a700 1 -- 172.21.15.118:0/2458831078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e0005040 con 0x7f27ec0f5410 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.452+0000 7f27f223a700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f27e0005010 tx=0x7f27e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e0009320 con 0x7f27ec0f5410 2024-02-23T00:26:36.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec060440 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27e0013380 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e0014e50 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec109b90 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.453+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27e00263a0 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.454+0000 7f27deffd700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 0x7f27d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.454+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec045ad0 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.454+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27e0040eb0 con 0x7f27ec0f5410 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.454+0000 7f27f1a39700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 0x7f27d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:36.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.454+0000 7f27f1a39700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 0x7f27d80323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f27e80096a0 tx=0x7f27e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:36.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.458+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27e0012920 con 0x7f27ec0f5410 2024-02-23T00:26:36.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.668+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27ec10a3b0 con 0x7f27ec0f5410 2024-02-23T00:26:36.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.668+0000 7f27deffd700 1 -- 172.21.15.118:0/2458831078 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27e0012330 con 0x7f27ec0f5410 2024-02-23T00:26:36.670 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:36.670 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:36.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 msgr2=0x7f27d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:36.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 0x7f27d80323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f27e80096a0 tx=0x7f27e8006b40).stop 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 msgr2=0x7f27ec1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f27e0005010 tx=0x7f27e000b040).stop 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 shutdown_connections 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f27d802ff10 0x7f27d80323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 --2- 172.21.15.118:0/2458831078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27ec0f5410 0x7f27ec1094e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 >> 172.21.15.118:0/2458831078 conn(0x7f27ec0f0b70 msgr2=0x7f27ec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 shutdown_connections 2024-02-23T00:26:36.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:36.670+0000 7f27f449e700 1 -- 172.21.15.118:0/2458831078 wait complete. 2024-02-23T00:26:36.673 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:37.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:36.669414+0000 mon.smithi090 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.118:0/2458831078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:38.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:37.141116+0000 mgr.smithi090.pxxmlz (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:38.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:38.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:39.141350+0000 mgr.smithi090.pxxmlz (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:40.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- 172.21.15.118:0/880848082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 msgr2=0x7f6af40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:40.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/880848082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af40f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6adc009fb0 tx=0x7f6adc009b30).stop 2024-02-23T00:26:40.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- 172.21.15.118:0/880848082 shutdown_connections 2024-02-23T00:26:40.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/880848082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af40f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- 172.21.15.118:0/880848082 >> 172.21.15.118:0/880848082 conn(0x7f6af40f1c70 msgr2=0x7f6af40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- 172.21.15.118:0/880848082 shutdown_connections 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- 172.21.15.118:0/880848082 wait complete. 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 Processor -- start 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.448+0000 7f6af8cd8700 1 -- start start 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af8cd8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af8cd8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af4106cd0 con 0x7f6af40f6510 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33756/0 (socket says 172.21.15.118:33756) 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af259c700 1 -- 172.21.15.118:0/3797194714 learned_addr learned my addr 172.21.15.118:0/3797194714 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af259c700 1 -- 172.21.15.118:0/3797194714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc005040 con 0x7f6af40f6510 2024-02-23T00:26:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af259c700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6adc0089f0 tx=0x7f6adc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:40.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6adc009320 con 0x7f6af40f6510 2024-02-23T00:26:40.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af4106ed0 con 0x7f6af40f6510 2024-02-23T00:26:40.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6adc013380 con 0x7f6af40f6510 2024-02-23T00:26:40.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6adc01ed80 con 0x7f6af40f6510 2024-02-23T00:26:40.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.449+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af4107360 con 0x7f6af40f6510 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.450+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6adc01e420 con 0x7f6af40f6510 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.450+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af4045ad0 con 0x7f6af40f6510 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.450+0000 7f6aeaffd700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 0x7f6ae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.450+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6adc041150 con 0x7f6af40f6510 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.451+0000 7f6af1d9b700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 0x7f6ae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:40.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.451+0000 7f6af1d9b700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 0x7f6ae0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6ae40096a0 tx=0x7f6ae4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:40.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.455+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6adc0095f0 con 0x7f6af40f6510 2024-02-23T00:26:40.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.661+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6af4108210 con 0x7f6af40f6510 2024-02-23T00:26:40.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.661+0000 7f6aeaffd700 1 -- 172.21.15.118:0/3797194714 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6adc004490 con 0x7f6af40f6510 2024-02-23T00:26:40.663 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:40.663 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:40.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 msgr2=0x7f6ae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:40.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 0x7f6ae0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6ae40096a0 tx=0x7f6ae4006b40).stop 2024-02-23T00:26:40.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 msgr2=0x7f6af41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:40.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6adc0089f0 tx=0x7f6adc00b040).stop 2024-02-23T00:26:40.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 shutdown_connections 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6ae002ff60 0x7f6ae0032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 --2- 172.21.15.118:0/3797194714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6af40f6510 0x7f6af41067e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 >> 172.21.15.118:0/3797194714 conn(0x7f6af40f1c70 msgr2=0x7f6af40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 shutdown_connections 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:40.663+0000 7f6af8cd8700 1 -- 172.21.15.118:0/3797194714 wait complete. 2024-02-23T00:26:40.665 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:40.662269+0000 mon.smithi090 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.118:0/3797194714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:41.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:41.141627+0000 mgr.smithi090.pxxmlz (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:42.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:42.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:43.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:43.141963+0000 mgr.smithi090.pxxmlz (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 -- 172.21.15.118:0/1190938897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 msgr2=0x7f29100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 --2- 172.21.15.118:0/1190938897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29100f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2900009fb0 tx=0x7f2900009b30).stop 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 -- 172.21.15.118:0/1190938897 shutdown_connections 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 --2- 172.21.15.118:0/1190938897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29100f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 -- 172.21.15.118:0/1190938897 >> 172.21.15.118:0/1190938897 conn(0x7f29100f0b70 msgr2=0x7f29100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 -- 172.21.15.118:0/1190938897 shutdown_connections 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 -- 172.21.15.118:0/1190938897 wait complete. 2024-02-23T00:26:44.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.488+0000 7f291651b700 1 Processor -- start 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f291651b700 1 -- start start 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f291651b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f291651b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2910105bd0 con 0x7f29100f5410 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f290ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f290ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:52768/0 (socket says 172.21.15.118:52768) 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f290ffff700 1 -- 172.21.15.118:0/3110953389 learned_addr learned my addr 172.21.15.118:0/3110953389 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f290ffff700 1 -- 172.21.15.118:0/3110953389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900005040 con 0x7f29100f5410 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.489+0000 7f290ffff700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f29000089f0 tx=0x7f290000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2900009320 con 0x7f29100f5410 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2910105dd0 con 0x7f29100f5410 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2900013380 con 0x7f29100f5410 2024-02-23T00:26:44.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290001ed80 con 0x7f29100f5410 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2910106260 con 0x7f29100f5410 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2910045ad0 con 0x7f29100f5410 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.490+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f290001e420 con 0x7f29100f5410 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.491+0000 7f290cff9700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 0x7f28f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.491+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29000410b0 con 0x7f29100f5410 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.491+0000 7f290f7fe700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 0x7f28f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.491+0000 7f290f7fe700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 0x7f28f8032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f29040096a0 tx=0x7f2904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:44.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.495+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29000095f0 con 0x7f29100f5410 2024-02-23T00:26:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.707+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29101070b0 con 0x7f29100f5410 2024-02-23T00:26:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.708+0000 7f290cff9700 1 -- 172.21.15.118:0/3110953389 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f290001cb10 con 0x7f29100f5410 2024-02-23T00:26:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:44.710 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 msgr2=0x7f28f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 0x7f28f8032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f29040096a0 tx=0x7f2904006b40).stop 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 msgr2=0x7f29101056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f29000089f0 tx=0x7f290000b040).stop 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 shutdown_connections 2024-02-23T00:26:44.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f28f802ff60 0x7f28f8032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:44.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 --2- 172.21.15.118:0/3110953389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29100f5410 0x7f29101056e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:44.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 >> 172.21.15.118:0/3110953389 conn(0x7f29100f0b70 msgr2=0x7f29100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:44.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 shutdown_connections 2024-02-23T00:26:44.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:44.710+0000 7f291651b700 1 -- 172.21.15.118:0/3110953389 wait complete. 2024-02-23T00:26:44.712 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:45.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:44.709011+0000 mon.smithi090 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.118:0/3110953389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:46.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:45.142279+0000 mgr.smithi090.pxxmlz (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:46.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:46.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:47.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:47.142670+0000 mgr.smithi090.pxxmlz (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 -- 172.21.15.118:0/4072581899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 msgr2=0x7f48c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 --2- 172.21.15.118:0/4072581899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f48b4009960 tx=0x7f48b4004d10).stop 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 -- 172.21.15.118:0/4072581899 shutdown_connections 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 --2- 172.21.15.118:0/4072581899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 -- 172.21.15.118:0/4072581899 >> 172.21.15.118:0/4072581899 conn(0x7f48c40f0b70 msgr2=0x7f48c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:48.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.514+0000 7f48c940a700 1 -- 172.21.15.118:0/4072581899 shutdown_connections 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.515+0000 7f48c940a700 1 -- 172.21.15.118:0/4072581899 wait complete. 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.515+0000 7f48c940a700 1 Processor -- start 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.515+0000 7f48c940a700 1 -- start start 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.515+0000 7f48c940a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.515+0000 7f48c940a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4105bd0 con 0x7f48c40f5410 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:52792/0 (socket says 172.21.15.118:52792) 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c2ffd700 1 -- 172.21.15.118:0/1016892369 learned_addr learned my addr 172.21.15.118:0/1016892369 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c2ffd700 1 -- 172.21.15.118:0/1016892369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b4009640 con 0x7f48c40f5410 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c2ffd700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f48b4004b70 tx=0x7f48b4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:48.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b4018070 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c4105dd0 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48b400b970 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b401c560 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.516+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c4106260 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.517+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48b4013630 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.517+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c4045ad0 con 0x7f48c40f5410 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.517+0000 7f48ab7fe700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:48.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.517+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48b40412b0 con 0x7f48c40f5410 2024-02-23T00:26:48.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.517+0000 7f48c27fc700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:48.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.518+0000 7f48c27fc700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 0x7f48ac032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f48b80096a0 tx=0x7f48b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:48.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.522+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48b40139f0 con 0x7f48c40f5410 2024-02-23T00:26:48.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.724+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48c4107110 con 0x7f48c40f5410 2024-02-23T00:26:48.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.725+0000 7f48ab7fe700 1 -- 172.21.15.118:0/1016892369 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48b4012380 con 0x7f48c40f5410 2024-02-23T00:26:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:48.727 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:48.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 msgr2=0x7f48ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:48.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 0x7f48ac032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f48b80096a0 tx=0x7f48b8006b40).stop 2024-02-23T00:26:48.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 msgr2=0x7f48c41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:48.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f48b4004b70 tx=0x7f48b4004d10).stop 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 shutdown_connections 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f48ac02ff60 0x7f48ac032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 --2- 172.21.15.118:0/1016892369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48c40f5410 0x7f48c41056e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 >> 172.21.15.118:0/1016892369 conn(0x7f48c40f0b70 msgr2=0x7f48c40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 shutdown_connections 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:48.727+0000 7f48c940a700 1 -- 172.21.15.118:0/1016892369 wait complete. 2024-02-23T00:26:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:49.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:48 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:48.725725+0000 mon.smithi090 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.118:0/1016892369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:50.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:49.143052+0000 mgr.smithi090.pxxmlz (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:50.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:50.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:51.143375+0000 mgr.smithi090.pxxmlz (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 -- 172.21.15.118:0/1005729995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 msgr2=0x7f05a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 --2- 172.21.15.118:0/1005729995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a80f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f05a4009fb0 tx=0x7f05a4009b30).stop 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 -- 172.21.15.118:0/1005729995 shutdown_connections 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 --2- 172.21.15.118:0/1005729995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a80f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 -- 172.21.15.118:0/1005729995 >> 172.21.15.118:0/1005729995 conn(0x7f05a80f0b70 msgr2=0x7f05a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 -- 172.21.15.118:0/1005729995 shutdown_connections 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 -- 172.21.15.118:0/1005729995 wait complete. 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.432+0000 7f05afe0f700 1 Processor -- start 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05afe0f700 1 -- start start 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05afe0f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05afe0f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8106cd0 con 0x7f05a80f5410 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05adbab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05adbab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:52818/0 (socket says 172.21.15.118:52818) 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05adbab700 1 -- 172.21.15.118:0/3010715203 learned_addr learned my addr 172.21.15.118:0/3010715203 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05adbab700 1 -- 172.21.15.118:0/3010715203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4005040 con 0x7f05a80f5410 2024-02-23T00:26:52.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05adbab700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f05a4005010 tx=0x7f05a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a4009320 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.433+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a8106ed0 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05a4013380 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05a4014e50 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a8107360 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a8045ad0 con 0x7f05a80f5410 2024-02-23T00:26:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05a40095f0 con 0x7f05a80f5410 2024-02-23T00:26:52.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.434+0000 7f059e7fc700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 0x7f05940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:52.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.435+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05a4040e90 con 0x7f05a80f5410 2024-02-23T00:26:52.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.435+0000 7f05ad3aa700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 0x7f05940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:52.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.435+0000 7f05ad3aa700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 0x7f05940323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.439+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05a4014420 con 0x7f05a80f5410 2024-02-23T00:26:52.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.644+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05a8108210 con 0x7f05a80f5410 2024-02-23T00:26:52.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.645+0000 7f059e7fc700 1 -- 172.21.15.118:0/3010715203 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f05a4010230 con 0x7f05a80f5410 2024-02-23T00:26:52.647 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:52.647 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 msgr2=0x7f05940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 0x7f05940323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).stop 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 msgr2=0x7f05a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f05a4005010 tx=0x7f05a400b040).stop 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 shutdown_connections 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f059402ff10 0x7f05940323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 --2- 172.21.15.118:0/3010715203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05a80f5410 0x7f05a81067e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 >> 172.21.15.118:0/3010715203 conn(0x7f05a80f0b70 msgr2=0x7f05a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 shutdown_connections 2024-02-23T00:26:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:52.646+0000 7f05afe0f700 1 -- 172.21.15.118:0/3010715203 wait complete. 2024-02-23T00:26:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:53.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:52 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:52.645731+0000 mon.smithi090 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.118:0/3010715203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:54.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:53.143699+0000 mgr.smithi090.pxxmlz (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:54.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:54.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:55.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:55.143996+0000 mgr.smithi090.pxxmlz (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- 172.21.15.118:0/2393173538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 msgr2=0x7f730c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 --2- 172.21.15.118:0/2393173538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f72f4009fb0 tx=0x7f72f4009b30).stop 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- 172.21.15.118:0/2393173538 shutdown_connections 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 --2- 172.21.15.118:0/2393173538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- 172.21.15.118:0/2393173538 >> 172.21.15.118:0/2393173538 conn(0x7f730c0f0b70 msgr2=0x7f730c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- 172.21.15.118:0/2393173538 shutdown_connections 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- 172.21.15.118:0/2393173538 wait complete. 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 Processor -- start 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.395+0000 7f73128c7700 1 -- start start 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f73128c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:56.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f73128c7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c107330 con 0x7f730c0f5410 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f730bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f730bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57982/0 (socket says 172.21.15.118:57982) 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f730bfff700 1 -- 172.21.15.118:0/2013177987 learned_addr learned my addr 172.21.15.118:0/2013177987 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f730bfff700 1 -- 172.21.15.118:0/2013177987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4005040 con 0x7f730c0f5410 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f730bfff700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f72f4005010 tx=0x7f72f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f4009320 con 0x7f730c0f5410 2024-02-23T00:26:56.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c060220 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72f4013380 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f4014e50 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.396+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c10be30 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.397+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72f40095f0 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.397+0000 7f7308ff9700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 0x7f72f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.397+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c045ad0 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.397+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72f4040ef0 con 0x7f730c0f5410 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.398+0000 7f730b7fe700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 0x7f72f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:56.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.398+0000 7f730b7fe700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 0x7f72f80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f72fc0096a0 tx=0x7f72fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:56.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.402+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72f4014420 con 0x7f730c0f5410 2024-02-23T00:26:56.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.612+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f730c10c5f0 con 0x7f730c0f5410 2024-02-23T00:26:56.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.613+0000 7f7308ff9700 1 -- 172.21.15.118:0/2013177987 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72f4010230 con 0x7f730c0f5410 2024-02-23T00:26:56.614 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:26:56.614 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:26:56.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 msgr2=0x7f72f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 0x7f72f80323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f72fc0096a0 tx=0x7f72fc006b40).stop 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 msgr2=0x7f730c10b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f72f4005010 tx=0x7f72f400b040).stop 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 shutdown_connections 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f72f802ff10 0x7f72f80323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 --2- 172.21.15.118:0/2013177987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f730c0f5410 0x7f730c10b940 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 >> 172.21.15.118:0/2013177987 conn(0x7f730c0f0b70 msgr2=0x7f730c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 shutdown_connections 2024-02-23T00:26:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:26:56.614+0000 7f73128c7700 1 -- 172.21.15.118:0/2013177987 wait complete. 2024-02-23T00:26:56.617 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:26:57.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:26:56.613597+0000 mon.smithi090 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.118:0/2013177987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:26:58.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:57.144295+0000 mgr.smithi090.pxxmlz (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:26:58.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:26:58.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:26:59.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:26:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:26:59.144644+0000 mgr.smithi090.pxxmlz (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 -- 172.21.15.118:0/312434057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 msgr2=0x7fb88c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 --2- 172.21.15.118:0/312434057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c0f8f80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb87c009fb0 tx=0x7fb87c009b30).stop 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 -- 172.21.15.118:0/312434057 shutdown_connections 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 --2- 172.21.15.118:0/312434057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c0f8f80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 -- 172.21.15.118:0/312434057 >> 172.21.15.118:0/312434057 conn(0x7fb88c0f09a0 msgr2=0x7fb88c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 -- 172.21.15.118:0/312434057 shutdown_connections 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.248+0000 7fb8921c9700 1 -- 172.21.15.118:0/312434057 wait complete. 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8921c9700 1 Processor -- start 2024-02-23T00:27:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8921c9700 1 -- start start 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8921c9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8921c9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c106b00 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8911c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8911c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58012/0 (socket says 172.21.15.118:58012) 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8911c7700 1 -- 172.21.15.118:0/2630158069 learned_addr learned my addr 172.21.15.118:0/2630158069 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8911c7700 1 -- 172.21.15.118:0/2630158069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c005040 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8911c7700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb87c0089f0 tx=0x7fb87c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87c009320 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.249+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c106d00 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.250+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb87c013380 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.250+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87c01ed80 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.250+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c107190 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.250+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb87c014420 con 0x7fb88c060180 2024-02-23T00:27:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.251+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c045ad0 con 0x7fb88c060180 2024-02-23T00:27:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.251+0000 7fb881ffb700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 0x7fb878032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.251+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb87c0411e0 con 0x7fb88c060180 2024-02-23T00:27:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.251+0000 7fb8909c6700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 0x7fb878032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.251+0000 7fb8909c6700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 0x7fb878032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb8880096a0 tx=0x7fb888006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.255+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb87c00fcc0 con 0x7fb88c060180 2024-02-23T00:27:00.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.462+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb88c107fe0 con 0x7fb88c060180 2024-02-23T00:27:00.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.463+0000 7fb881ffb700 1 -- 172.21.15.118:0/2630158069 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb87c00fae0 con 0x7fb88c060180 2024-02-23T00:27:00.464 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:00.465 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:00.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 msgr2=0x7fb878032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:00.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 0x7fb878032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb8880096a0 tx=0x7fb888006b40).stop 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 msgr2=0x7fb88c106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb87c0089f0 tx=0x7fb87c00b040).stop 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 shutdown_connections 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb87802ff60 0x7fb878032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 --2- 172.21.15.118:0/2630158069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb88c060180 0x7fb88c106610 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 >> 172.21.15.118:0/2630158069 conn(0x7fb88c0f09a0 msgr2=0x7fb88c0f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 shutdown_connections 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:00.465+0000 7fb8921c9700 1 -- 172.21.15.118:0/2630158069 wait complete. 2024-02-23T00:27:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:00.464002+0000 mon.smithi090 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.118:0/2630158069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:01.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:01.144968+0000 mgr.smithi090.pxxmlz (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:02.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:02.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:03.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:03.145186+0000 mgr.smithi090.pxxmlz (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:04.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 -- 172.21.15.118:0/969930640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 msgr2=0x7fc5480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 --2- 172.21.15.118:0/969930640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5480f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc538009fb0 tx=0x7fc538009b30).stop 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 -- 172.21.15.118:0/969930640 shutdown_connections 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 --2- 172.21.15.118:0/969930640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5480f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 -- 172.21.15.118:0/969930640 >> 172.21.15.118:0/969930640 conn(0x7fc5480f0b70 msgr2=0x7fc5480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 -- 172.21.15.118:0/969930640 shutdown_connections 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.013+0000 7fc5501d2700 1 -- 172.21.15.118:0/969930640 wait complete. 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc5501d2700 1 Processor -- start 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc5501d2700 1 -- start start 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc5501d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc5501d2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc548060240 con 0x7fc5480f5410 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc54df6e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc54df6e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58036/0 (socket says 172.21.15.118:58036) 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc54df6e700 1 -- 172.21.15.118:0/2551170978 learned_addr learned my addr 172.21.15.118:0/2551170978 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc54df6e700 1 -- 172.21.15.118:0/2551170978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538005040 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc54df6e700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc538005010 tx=0x7fc53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc538009320 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc548060440 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc538013380 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.014+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc538014e50 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.015+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc548109b90 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.015+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5380095f0 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.015+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc548045ad0 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.015+0000 7fc53e7fc700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 0x7fc5340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.015+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc538040e70 con 0x7fc5480f5410 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.016+0000 7fc54d76d700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 0x7fc5340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.016+0000 7fc54d76d700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 0x7fc5340323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc5440096a0 tx=0x7fc544006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:04.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.020+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc538014420 con 0x7fc5480f5410 2024-02-23T00:27:04.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.226+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc54810a3b0 con 0x7fc5480f5410 2024-02-23T00:27:04.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.227+0000 7fc53e7fc700 1 -- 172.21.15.118:0/2551170978 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc538010230 con 0x7fc5480f5410 2024-02-23T00:27:04.229 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:04.230 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 msgr2=0x7fc5340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 0x7fc5340323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc5440096a0 tx=0x7fc544006b40).stop 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 msgr2=0x7fc5481094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc538005010 tx=0x7fc53800b040).stop 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 shutdown_connections 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc53402ff10 0x7fc5340323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 --2- 172.21.15.118:0/2551170978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5480f5410 0x7fc5481094e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 >> 172.21.15.118:0/2551170978 conn(0x7fc5480f0b70 msgr2=0x7fc5480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:04.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 shutdown_connections 2024-02-23T00:27:04.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:04.228+0000 7fc5501d2700 1 -- 172.21.15.118:0/2551170978 wait complete. 2024-02-23T00:27:04.232 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:04.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:04.228674+0000 mon.smithi090 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.118:0/2551170978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:05.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:05.145431+0000 mgr.smithi090.pxxmlz (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:05.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:05.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:07.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:07.145700+0000 mgr.smithi090.pxxmlz (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:08.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.006+0000 7f361560a700 1 -- 172.21.15.118:0/595616251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 msgr2=0x7f36100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:08.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.006+0000 7f361560a700 1 --2- 172.21.15.118:0/595616251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36100f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3600009fb0 tx=0x7f3600009b30).stop 2024-02-23T00:27:08.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- 172.21.15.118:0/595616251 shutdown_connections 2024-02-23T00:27:08.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 --2- 172.21.15.118:0/595616251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36100f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:08.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- 172.21.15.118:0/595616251 >> 172.21.15.118:0/595616251 conn(0x7f36100f0b70 msgr2=0x7f36100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:08.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- 172.21.15.118:0/595616251 shutdown_connections 2024-02-23T00:27:08.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- 172.21.15.118:0/595616251 wait complete. 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 Processor -- start 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- start start 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.007+0000 7f361560a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3610060240 con 0x7f36100f5410 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f360effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f360effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36006/0 (socket says 172.21.15.118:36006) 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f360effd700 1 -- 172.21.15.118:0/3667517352 learned_addr learned my addr 172.21.15.118:0/3667517352 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f360effd700 1 -- 172.21.15.118:0/3667517352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3600005040 con 0x7f36100f5410 2024-02-23T00:27:08.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f360effd700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3600005010 tx=0x7f360000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3600009320 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3610060440 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3600013380 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3600014e50 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.008+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610109b90 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.009+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36000095f0 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.009+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610045ad0 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.009+0000 7f35f77fe700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 0x7f35f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.009+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3600040fa0 con 0x7f36100f5410 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.010+0000 7f360e7fc700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 0x7f35f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:08.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.010+0000 7f360e7fc700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 0x7f35f80323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f36040096a0 tx=0x7f3604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:08.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.015+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3600014420 con 0x7f36100f5410 2024-02-23T00:27:08.221 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.219+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f361010a3b0 con 0x7f36100f5410 2024-02-23T00:27:08.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.219+0000 7f35f77fe700 1 -- 172.21.15.118:0/3667517352 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3600010230 con 0x7f36100f5410 2024-02-23T00:27:08.222 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:08.222 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:08.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 msgr2=0x7f35f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 0x7f35f80323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f36040096a0 tx=0x7f3604006b40).stop 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 msgr2=0x7f36101094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3600005010 tx=0x7f360000b040).stop 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 shutdown_connections 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f35f802ff10 0x7f35f80323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 --2- 172.21.15.118:0/3667517352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36100f5410 0x7f36101094e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 >> 172.21.15.118:0/3667517352 conn(0x7f36100f0b70 msgr2=0x7f36100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 shutdown_connections 2024-02-23T00:27:08.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:08.221+0000 7f361560a700 1 -- 172.21.15.118:0/3667517352 wait complete. 2024-02-23T00:27:08.225 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:08.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:08.221284+0000 mon.smithi090 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.118:0/3667517352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:09.146047+0000 mgr.smithi090.pxxmlz (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:09.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:09.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:11.146344+0000 mgr.smithi090.pxxmlz (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/3413485174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 msgr2=0x7f3bb80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/3413485174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f90f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3ba8009fb0 tx=0x7f3ba8009b30).stop 2024-02-23T00:27:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/3413485174 shutdown_connections 2024-02-23T00:27:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/3413485174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f90f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/3413485174 >> 172.21.15.118:0/3413485174 conn(0x7f3bb80f0b70 msgr2=0x7f3bb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/3413485174 shutdown_connections 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.057+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/3413485174 wait complete. 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bbe3f3700 1 Processor -- start 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bbe3f3700 1 -- start start 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bbe3f3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bbe3f3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80f4ff0 con 0x7f3bb8060280 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bb7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bb7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36038/0 (socket says 172.21.15.118:36038) 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bb7fff700 1 -- 172.21.15.118:0/1472782457 learned_addr learned my addr 172.21.15.118:0/1472782457 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.058+0000 7f3bb7fff700 1 -- 172.21.15.118:0/1472782457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8005040 con 0x7f3bb8060280 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bb7fff700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3ba8009830 tx=0x7f3ba800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba8009320 con 0x7f3bb8060280 2024-02-23T00:27:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb80f5190 con 0x7f3bb8060280 2024-02-23T00:27:12.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ba8013380 con 0x7f3bb8060280 2024-02-23T00:27:12.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ba8014420 con 0x7f3bb8060280 2024-02-23T00:27:12.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.059+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb80f5620 con 0x7f3bb8060280 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.060+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ba801e420 con 0x7f3bb8060280 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.060+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8059100 con 0x7f3bb8060280 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.060+0000 7f3bb4ff9700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 0x7f3ba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.060+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ba8040e20 con 0x7f3bb8060280 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.060+0000 7f3bb77fe700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 0x7f3ba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.061+0000 7f3bb77fe700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 0x7f3ba4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3ba00096a0 tx=0x7f3ba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:12.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.065+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ba801c970 con 0x7f3bb8060280 2024-02-23T00:27:12.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.269+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bb8045ad0 con 0x7f3bb8060280 2024-02-23T00:27:12.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.270+0000 7f3bb4ff9700 1 -- 172.21.15.118:0/1472782457 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ba802c2d0 con 0x7f3bb8060280 2024-02-23T00:27:12.271 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:12.271 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 msgr2=0x7f3ba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 0x7f3ba4032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3ba00096a0 tx=0x7f3ba0006b40).stop 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 msgr2=0x7f3bb80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3ba8009830 tx=0x7f3ba800b040).stop 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 shutdown_connections 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ba402ff60 0x7f3ba4032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 --2- 172.21.15.118:0/1472782457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bb8060280 0x7f3bb80f6b70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.271+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 >> 172.21.15.118:0/1472782457 conn(0x7f3bb80f0b70 msgr2=0x7f3bb80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:12.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.272+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 shutdown_connections 2024-02-23T00:27:12.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:12.272+0000 7f3bbe3f3700 1 -- 172.21.15.118:0/1472782457 wait complete. 2024-02-23T00:27:12.274 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:12.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:12.270594+0000 mon.smithi090 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.118:0/1472782457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:13.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:13.146641+0000 mgr.smithi090.pxxmlz (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:13.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:13.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:15.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:15.146930+0000 mgr.smithi090.pxxmlz (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 -- 172.21.15.118:0/2129128358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 msgr2=0x7f34100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 --2- 172.21.15.118:0/2129128358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34100f66c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3404009fb0 tx=0x7f3404009b30).stop 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 -- 172.21.15.118:0/2129128358 shutdown_connections 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 --2- 172.21.15.118:0/2129128358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34100f66c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 -- 172.21.15.118:0/2129128358 >> 172.21.15.118:0/2129128358 conn(0x7f34100f1aa0 msgr2=0x7f34100f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 -- 172.21.15.118:0/2129128358 shutdown_connections 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.779+0000 7f34153a2700 1 -- 172.21.15.118:0/2129128358 wait complete. 2024-02-23T00:27:15.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f34153a2700 1 Processor -- start 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f34153a2700 1 -- start start 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f34153a2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f34153a2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3410106ac0 con 0x7f34100f62a0 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f340ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f340ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:38002/0 (socket says 172.21.15.118:38002) 2024-02-23T00:27:15.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.780+0000 7f340ffff700 1 -- 172.21.15.118:0/875185747 learned_addr learned my addr 172.21.15.118:0/875185747 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f340ffff700 1 -- 172.21.15.118:0/875185747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404005040 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f340ffff700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f34040089f0 tx=0x7f340400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3404009320 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3410106cc0 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3404013380 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3404014e20 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.781+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3410107150 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.782+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34040095f0 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.782+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410045ad0 con 0x7f34100f62a0 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.782+0000 7f340cff9700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 0x7f33f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.782+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3404040e90 con 0x7f34100f62a0 2024-02-23T00:27:15.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.782+0000 7f340f7fe700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 0x7f33f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.783+0000 7f340f7fe700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 0x7f33f80323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:15.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.787+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3404014420 con 0x7f34100f62a0 2024-02-23T00:27:15.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.990+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3410108000 con 0x7f34100f62a0 2024-02-23T00:27:15.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.991+0000 7f340cff9700 1 -- 172.21.15.118:0/875185747 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f340401d980 con 0x7f34100f62a0 2024-02-23T00:27:15.992 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:15.993 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:15.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 msgr2=0x7f33f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 0x7f33f80323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).stop 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 msgr2=0x7f34101065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f34040089f0 tx=0x7f340400b040).stop 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 shutdown_connections 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f33f802ff10 0x7f33f80323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 --2- 172.21.15.118:0/875185747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34100f62a0 0x7f34101065d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 >> 172.21.15.118:0/875185747 conn(0x7f34100f1aa0 msgr2=0x7f34100f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 shutdown_connections 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:15.992+0000 7f34153a2700 1 -- 172.21.15.118:0/875185747 wait complete. 2024-02-23T00:27:15.994 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:16.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:15.991739+0000 mon.smithi090 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.118:0/875185747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:17.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:17.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:17.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:17.147268+0000 mgr.smithi090.pxxmlz (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:19.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:19.147602+0000 mgr.smithi090.pxxmlz (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 -- 172.21.15.118:0/214868459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 msgr2=0x7ff3600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 --2- 172.21.15.118:0/214868459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff354009fb0 tx=0x7ff354009b30).stop 2024-02-23T00:27:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 -- 172.21.15.118:0/214868459 shutdown_connections 2024-02-23T00:27:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 --2- 172.21.15.118:0/214868459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 -- 172.21.15.118:0/214868459 >> 172.21.15.118:0/214868459 conn(0x7ff3600f0b70 msgr2=0x7ff3600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 -- 172.21.15.118:0/214868459 shutdown_connections 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.760+0000 7ff365179700 1 -- 172.21.15.118:0/214868459 wait complete. 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff365179700 1 Processor -- start 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff365179700 1 -- start start 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff365179700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff365179700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600fb960 con 0x7ff3600f5410 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff35ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff35ed9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:38030/0 (socket says 172.21.15.118:38030) 2024-02-23T00:27:19.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff35ed9d700 1 -- 172.21.15.118:0/937677194 learned_addr learned my addr 172.21.15.118:0/937677194 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff35ed9d700 1 -- 172.21.15.118:0/937677194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354005040 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff35ed9d700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff354005010 tx=0x7ff35400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.761+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff354009320 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3600fbb00 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff354013380 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff354014e50 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600fbf90 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3540263a0 con 0x7ff3600f5410 2024-02-23T00:27:19.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.762+0000 7ff3477fe700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 0x7ff3480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:19.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.763+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff354040fc0 con 0x7ff3600f5410 2024-02-23T00:27:19.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.763+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360059100 con 0x7ff3600f5410 2024-02-23T00:27:19.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.763+0000 7ff35e59c700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 0x7ff3480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:19.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.763+0000 7ff35e59c700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 0x7ff3480323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff3500096a0 tx=0x7ff350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:19.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.767+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff354012920 con 0x7ff3600f5410 2024-02-23T00:27:19.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.970+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3600fc690 con 0x7ff3600f5410 2024-02-23T00:27:19.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.971+0000 7ff3477fe700 1 -- 172.21.15.118:0/937677194 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff35402b020 con 0x7ff3600f5410 2024-02-23T00:27:19.973 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:19.973 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 msgr2=0x7ff3480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 0x7ff3480323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff3500096a0 tx=0x7ff350006b40).stop 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 msgr2=0x7ff3600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff354005010 tx=0x7ff35400b040).stop 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 shutdown_connections 2024-02-23T00:27:19.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff34802ff10 0x7ff3480323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 --2- 172.21.15.118:0/937677194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3600f5410 0x7ff3600fd4e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 >> 172.21.15.118:0/937677194 conn(0x7ff3600f0b70 msgr2=0x7ff36005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:19.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 shutdown_connections 2024-02-23T00:27:19.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:19.973+0000 7ff365179700 1 -- 172.21.15.118:0/937677194 wait complete. 2024-02-23T00:27:19.975 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:20.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:19.972006+0000 mon.smithi090 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.118:0/937677194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:21.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:21.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:21.147909+0000 mgr.smithi090.pxxmlz (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:23.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:23.148242+0000 mgr.smithi090.pxxmlz (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:23.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 -- 172.21.15.118:0/2368601024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 msgr2=0x7f469c0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 --2- 172.21.15.118:0/2368601024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c0f5750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4690009fb0 tx=0x7f4690009b30).stop 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 -- 172.21.15.118:0/2368601024 shutdown_connections 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 --2- 172.21.15.118:0/2368601024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c0f5750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 -- 172.21.15.118:0/2368601024 >> 172.21.15.118:0/2368601024 conn(0x7f469c0f0a90 msgr2=0x7f469c0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 -- 172.21.15.118:0/2368601024 shutdown_connections 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 -- 172.21.15.118:0/2368601024 wait complete. 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.648+0000 7f46a234d700 1 Processor -- start 2024-02-23T00:27:23.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f46a234d700 1 -- start start 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f46a234d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f46a234d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c105e70 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f469bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f469bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:38062/0 (socket says 172.21.15.118:38062) 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f469bfff700 1 -- 172.21.15.118:0/719304823 learned_addr learned my addr 172.21.15.118:0/719304823 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f469bfff700 1 -- 172.21.15.118:0/719304823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4690005040 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.649+0000 7f469bfff700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f46900089f0 tx=0x7f469000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4690009320 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c106070 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4690013380 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f469001ed80 con 0x7f469c0f5330 2024-02-23T00:27:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c106500 con 0x7f469c0f5330 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.650+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4690014420 con 0x7f469c0f5330 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.651+0000 7f4698ff9700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 0x7f4684032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.651+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4690040ec0 con 0x7f469c0f5330 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.651+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c045ad0 con 0x7f469c0f5330 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.651+0000 7f469b7fe700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 0x7f4684032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.651+0000 7f469b7fe700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 0x7f4684032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f468c0096a0 tx=0x7f468c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:23.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.655+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f469001c910 con 0x7f469c0f5330 2024-02-23T00:27:23.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.861+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f469c107180 con 0x7f469c0f5330 2024-02-23T00:27:23.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.862+0000 7f4698ff9700 1 -- 172.21.15.118:0/719304823 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f469001c320 con 0x7f469c0f5330 2024-02-23T00:27:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:23.864 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:23.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 msgr2=0x7f4684032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:23.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 0x7f4684032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f468c0096a0 tx=0x7f468c006b40).stop 2024-02-23T00:27:23.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 msgr2=0x7f469c105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:23.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f46900089f0 tx=0x7f469000b040).stop 2024-02-23T00:27:23.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 shutdown_connections 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f468402ff60 0x7f4684032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 --2- 172.21.15.118:0/719304823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f469c0f5330 0x7f469c105980 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 >> 172.21.15.118:0/719304823 conn(0x7f469c0f0a90 msgr2=0x7f469c0f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 shutdown_connections 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:23.864+0000 7f46a234d700 1 -- 172.21.15.118:0/719304823 wait complete. 2024-02-23T00:27:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:24.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:23.863057+0000 mon.smithi090 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.118:0/719304823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:25.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:25.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:25.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:25.148566+0000 mgr.smithi090.pxxmlz (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:26.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:25.245058+0000 mon.smithi090 (mon.0) 173 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:27:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:25.252823+0000 mon.smithi090 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 -- 172.21.15.118:0/3546051868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 msgr2=0x7f3a380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3546051868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a380f90f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3a2c009fb0 tx=0x7f3a2c009b30).stop 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 -- 172.21.15.118:0/3546051868 shutdown_connections 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3546051868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a380f90f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 -- 172.21.15.118:0/3546051868 >> 172.21.15.118:0/3546051868 conn(0x7f3a380f0b70 msgr2=0x7f3a380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 -- 172.21.15.118:0/3546051868 shutdown_connections 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 -- 172.21.15.118:0/3546051868 wait complete. 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.468+0000 7f3a3d784700 1 Processor -- start 2024-02-23T00:27:27.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a3d784700 1 -- start start 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a3d784700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a3d784700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38106c70 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a36ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58116/0 (socket says 172.21.15.118:58116) 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a36ffd700 1 -- 172.21.15.118:0/3569042501 learned_addr learned my addr 172.21.15.118:0/3569042501 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a36ffd700 1 -- 172.21.15.118:0/3569042501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c005040 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a36ffd700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3a2c0089f0 tx=0x7f3a2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.469+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a2c009320 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38106e70 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a2c013380 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a2c014e20 con 0x7f3a38060280 2024-02-23T00:27:27.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a38107300 con 0x7f3a38060280 2024-02-23T00:27:27.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a2c0095f0 con 0x7f3a38060280 2024-02-23T00:27:27.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38045ad0 con 0x7f3a38060280 2024-02-23T00:27:27.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.470+0000 7f3a1f7fe700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:27.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.471+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a2c040f50 con 0x7f3a38060280 2024-02-23T00:27:27.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.471+0000 7f3a367fc700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:27.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.471+0000 7f3a367fc700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 0x7f3a200323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3a280096a0 tx=0x7f3a28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:27.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.475+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a2c014420 con 0x7f3a38060280 2024-02-23T00:27:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:27.148805+0000 mgr.smithi090.pxxmlz (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:27.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.677+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a381081b0 con 0x7f3a38060280 2024-02-23T00:27:27.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.678+0000 7f3a1f7fe700 1 -- 172.21.15.118:0/3569042501 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a2c01d980 con 0x7f3a38060280 2024-02-23T00:27:27.680 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:27.680 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:27.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 msgr2=0x7f3a200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:27.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 0x7f3a200323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3a280096a0 tx=0x7f3a28006b40).stop 2024-02-23T00:27:27.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 msgr2=0x7f3a38106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:27.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3a2c0089f0 tx=0x7f3a2c00b040).stop 2024-02-23T00:27:27.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 shutdown_connections 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3a2002ff10 0x7f3a200323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 --2- 172.21.15.118:0/3569042501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a38060280 0x7f3a38106780 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 >> 172.21.15.118:0/3569042501 conn(0x7f3a380f0b70 msgr2=0x7f3a380f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 shutdown_connections 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:27.679+0000 7f3a3d784700 1 -- 172.21.15.118:0/3569042501 wait complete. 2024-02-23T00:27:27.682 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:28.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:27.678791+0000 mon.smithi090 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.118:0/3569042501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:29.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:29.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:29.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:29.149182+0000 mgr.smithi090.pxxmlz (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- 172.21.15.118:0/594298113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 msgr2=0x7f9f580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 --2- 172.21.15.118:0/594298113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f580f55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f9f4c009fb0 tx=0x7f9f4c009b30).stop 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- 172.21.15.118:0/594298113 shutdown_connections 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 --2- 172.21.15.118:0/594298113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f580f55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- 172.21.15.118:0/594298113 >> 172.21.15.118:0/594298113 conn(0x7f9f580f09a0 msgr2=0x7f9f580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:31.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- 172.21.15.118:0/594298113 shutdown_connections 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- 172.21.15.118:0/594298113 wait complete. 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 Processor -- start 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.402+0000 7f9f57fff700 1 -- start start 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f57fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f57fff700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58106ac0 con 0x7f9f580f51a0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f56ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:58142/0 (socket says 172.21.15.118:58142) 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f56ffd700 1 -- 172.21.15.118:0/2953302527 learned_addr learned my addr 172.21.15.118:0/2953302527 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f56ffd700 1 -- 172.21.15.118:0/2953302527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c005040 con 0x7f9f580f51a0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f56ffd700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9f4c0089f0 tx=0x7f9f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c009320 con 0x7f9f580f51a0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.403+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58106cc0 con 0x7f9f580f51a0 2024-02-23T00:27:31.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f4c013380 con 0x7f9f580f51a0 2024-02-23T00:27:31.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4c014e20 con 0x7f9f580f51a0 2024-02-23T00:27:31.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f58107150 con 0x7f9f580f51a0 2024-02-23T00:27:31.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f4c014420 con 0x7f9f580f51a0 2024-02-23T00:27:31.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f58045ad0 con 0x7f9f580f51a0 2024-02-23T00:27:31.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f3f7fe700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 0x7f9f400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:31.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.404+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f4c040fd0 con 0x7f9f580f51a0 2024-02-23T00:27:31.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.405+0000 7f9f567fc700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 0x7f9f400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:31.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.405+0000 7f9f567fc700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 0x7f9f400323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9f480096a0 tx=0x7f9f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:31.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.409+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f4c0095f0 con 0x7f9f580f51a0 2024-02-23T00:27:31.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.614+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f580fad30 con 0x7f9f580f51a0 2024-02-23T00:27:31.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.615+0000 7f9f3f7fe700 1 -- 172.21.15.118:0/2953302527 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f4c01db70 con 0x7f9f580f51a0 2024-02-23T00:27:31.616 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:31.617 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 msgr2=0x7f9f400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 0x7f9f400323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9f480096a0 tx=0x7f9f48006b40).stop 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 msgr2=0x7f9f581065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9f4c0089f0 tx=0x7f9f4c00b040).stop 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 shutdown_connections 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.616+0000 7f9f57fff700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9f4002ff10 0x7f9f400323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.617+0000 7f9f57fff700 1 --2- 172.21.15.118:0/2953302527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f580f51a0 0x7f9f581065d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:31.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.617+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 >> 172.21.15.118:0/2953302527 conn(0x7f9f580f09a0 msgr2=0x7f9f5805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:31.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.617+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 shutdown_connections 2024-02-23T00:27:31.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:31.617+0000 7f9f57fff700 1 -- 172.21.15.118:0/2953302527 wait complete. 2024-02-23T00:27:31.619 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:31.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:31.149412+0000 mgr.smithi090.pxxmlz (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:32.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:31.615835+0000 mon.smithi090 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.118:0/2953302527' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:33.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:33.380 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:33.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:33.149643+0000 mgr.smithi090.pxxmlz (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:35.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 -- 172.21.15.118:0/4241516657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 msgr2=0x7f5a980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/4241516657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a980f66c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5a88009fb0 tx=0x7f5a88009b30).stop 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 -- 172.21.15.118:0/4241516657 shutdown_connections 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/4241516657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a980f66c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 -- 172.21.15.118:0/4241516657 >> 172.21.15.118:0/4241516657 conn(0x7f5a980f1aa0 msgr2=0x7f5a980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 -- 172.21.15.118:0/4241516657 shutdown_connections 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.323+0000 7f5a9e86b700 1 -- 172.21.15.118:0/4241516657 wait complete. 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9e86b700 1 Processor -- start 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9e86b700 1 -- start start 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9e86b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9e86b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98106ac0 con 0x7f5a980f62a0 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9d869700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9d869700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57432/0 (socket says 172.21.15.118:57432) 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.324+0000 7f5a9d869700 1 -- 172.21.15.118:0/2021360377 learned_addr learned my addr 172.21.15.118:0/2021360377 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:35.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a9d869700 1 -- 172.21.15.118:0/2021360377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a88005040 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a9d869700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5a88009830 tx=0x7f5a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a88009320 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a88013380 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a98106cc0 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a88014420 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.325+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a98107150 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a8801e420 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a98045ad0 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a8e7fc700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a88040ef0 con 0x7f5a980f62a0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a9d068700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:35.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.326+0000 7f5a9d068700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 0x7f5a84032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5a940096a0 tx=0x7f5a94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:35.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.330+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a8801cb10 con 0x7f5a980f62a0 2024-02-23T00:27:35.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:35.149982+0000 mgr.smithi090.pxxmlz (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:35.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.533+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a980fbe30 con 0x7f5a980f62a0 2024-02-23T00:27:35.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.534+0000 7f5a8e7fc700 1 -- 172.21.15.118:0/2021360377 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a88004490 con 0x7f5a980f62a0 2024-02-23T00:27:35.535 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:35.536 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 msgr2=0x7f5a84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 0x7f5a84032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5a940096a0 tx=0x7f5a94006b40).stop 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 msgr2=0x7f5a981065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5a88009830 tx=0x7f5a8800b040).stop 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 shutdown_connections 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f5a8402ff60 0x7f5a84032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 --2- 172.21.15.118:0/2021360377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a980f62a0 0x7f5a981065d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 >> 172.21.15.118:0/2021360377 conn(0x7f5a980f1aa0 msgr2=0x7f5a980f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 shutdown_connections 2024-02-23T00:27:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:35.535+0000 7f5a9e86b700 1 -- 172.21.15.118:0/2021360377 wait complete. 2024-02-23T00:27:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:36.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:35.534562+0000 mon.smithi090 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.118:0/2021360377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:37.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:37.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:37.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:37.150313+0000 mgr.smithi090.pxxmlz (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:39.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 -- 172.21.15.118:0/116264284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 msgr2=0x7f47480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 --2- 172.21.15.118:0/116264284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47480f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4740009960 tx=0x7f4740004d10).stop 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 -- 172.21.15.118:0/116264284 shutdown_connections 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 --2- 172.21.15.118:0/116264284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47480f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 -- 172.21.15.118:0/116264284 >> 172.21.15.118:0/116264284 conn(0x7f47480f1c70 msgr2=0x7f47480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.018+0000 7f474cba9700 1 -- 172.21.15.118:0/116264284 shutdown_connections 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474cba9700 1 -- 172.21.15.118:0/116264284 wait complete. 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474cba9700 1 Processor -- start 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474cba9700 1 -- start start 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474cba9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474cba9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748106cd0 con 0x7f47480f6510 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57460/0 (socket says 172.21.15.118:57460) 2024-02-23T00:27:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.019+0000 7f474659c700 1 -- 172.21.15.118:0/1713868379 learned_addr learned my addr 172.21.15.118:0/1713868379 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f474659c700 1 -- 172.21.15.118:0/1713868379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4740009640 con 0x7f47480f6510 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f474659c700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4740004b70 tx=0x7f4740004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4740018070 con 0x7f47480f6510 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4748106ed0 con 0x7f47480f6510 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f474000b970 con 0x7f47480f6510 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474001c560 con 0x7f47480f6510 2024-02-23T00:27:39.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.020+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4748107360 con 0x7f47480f6510 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4740013630 con 0x7f47480f6510 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f4736ffd700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 0x7f4730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748045ad0 con 0x7f47480f6510 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47400412d0 con 0x7f47480f6510 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f4745d9b700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 0x7f4730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:39.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.021+0000 7f4745d9b700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 0x7f4730032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f473c0096a0 tx=0x7f473c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:39.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.025+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4740012560 con 0x7f47480f6510 2024-02-23T00:27:39.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.231+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4748108210 con 0x7f47480f6510 2024-02-23T00:27:39.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.232+0000 7f4736ffd700 1 -- 172.21.15.118:0/1713868379 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4740012380 con 0x7f47480f6510 2024-02-23T00:27:39.234 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:39.234 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 msgr2=0x7f4730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 0x7f4730032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f473c0096a0 tx=0x7f473c006b40).stop 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 msgr2=0x7f47481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4740004b70 tx=0x7f4740004d10).stop 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 shutdown_connections 2024-02-23T00:27:39.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f473002ff60 0x7f4730032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:39.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 --2- 172.21.15.118:0/1713868379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47480f6510 0x7f47481067e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:39.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 >> 172.21.15.118:0/1713868379 conn(0x7f47480f1c70 msgr2=0x7f47480f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:39.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 shutdown_connections 2024-02-23T00:27:39.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:39.234+0000 7f474cba9700 1 -- 172.21.15.118:0/1713868379 wait complete. 2024-02-23T00:27:39.236 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:39.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:39.150584+0000 mgr.smithi090.pxxmlz (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:39.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:39.233019+0000 mon.smithi090 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.118:0/1713868379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:40.904 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:40.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:41.150823+0000 mgr.smithi090.pxxmlz (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:42.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 -- 172.21.15.118:0/327422648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 msgr2=0x7f6d540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:42.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 --2- 172.21.15.118:0/327422648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d540f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f6d50009fb0 tx=0x7f6d50009b30).stop 2024-02-23T00:27:42.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 -- 172.21.15.118:0/327422648 shutdown_connections 2024-02-23T00:27:42.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 --2- 172.21.15.118:0/327422648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d540f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:42.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 -- 172.21.15.118:0/327422648 >> 172.21.15.118:0/327422648 conn(0x7f6d540f0b70 msgr2=0x7f6d540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 -- 172.21.15.118:0/327422648 shutdown_connections 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.991+0000 7f6d5c691700 1 -- 172.21.15.118:0/327422648 wait complete. 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5c691700 1 Processor -- start 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5c691700 1 -- start start 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5c691700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5c691700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54108ed0 con 0x7f6d540f5410 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5a42d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5a42d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57478/0 (socket says 172.21.15.118:57478) 2024-02-23T00:27:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5a42d700 1 -- 172.21.15.118:0/1937801238 learned_addr learned my addr 172.21.15.118:0/1937801238 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5a42d700 1 -- 172.21.15.118:0/1937801238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50005040 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.992+0000 7f6d5a42d700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f6d500089f0 tx=0x7f6d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d50009320 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d50013380 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d541090d0 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d5001ed80 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d54109560 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d5001e420 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.993+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d54045ad0 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.994+0000 7f6d4affd700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.994+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d500410f0 con 0x7f6d540f5410 2024-02-23T00:27:42.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.994+0000 7f6d59c2c700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:42.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.994+0000 7f6d59c2c700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 0x7f6d40032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f6d440096a0 tx=0x7f6d44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:42.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:42.998+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d5001ccf0 con 0x7f6d540f5410 2024-02-23T00:27:43.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.203+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d5410a3b0 con 0x7f6d540f5410 2024-02-23T00:27:43.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.203+0000 7f6d4affd700 1 -- 172.21.15.118:0/1937801238 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d5001cb10 con 0x7f6d540f5410 2024-02-23T00:27:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:43.205 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:43.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 msgr2=0x7f6d40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 0x7f6d40032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f6d440096a0 tx=0x7f6d44006b40).stop 2024-02-23T00:27:43.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 msgr2=0x7f6d541089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f6d500089f0 tx=0x7f6d5000b040).stop 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 shutdown_connections 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 --2- 172.21.15.118:0/1937801238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d540f5410 0x7f6d541089e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 >> 172.21.15.118:0/1937801238 conn(0x7f6d540f0b70 msgr2=0x7f6d5405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 shutdown_connections 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:43.205+0000 7f6d5c691700 1 -- 172.21.15.118:0/1937801238 wait complete. 2024-02-23T00:27:43.207 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:43.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:43.151101+0000 mgr.smithi090.pxxmlz (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:43.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:43.204171+0000 mon.smithi090 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.118:0/1937801238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:44.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:44.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:45.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:45.151377+0000 mgr.smithi090.pxxmlz (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:46.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 -- 172.21.15.118:0/3525755471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 msgr2=0x7f50e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 --2- 172.21.15.118:0/3525755471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e80f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f50dc009fb0 tx=0x7f50dc009b30).stop 2024-02-23T00:27:46.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 -- 172.21.15.118:0/3525755471 shutdown_connections 2024-02-23T00:27:46.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 --2- 172.21.15.118:0/3525755471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e80f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 -- 172.21.15.118:0/3525755471 >> 172.21.15.118:0/3525755471 conn(0x7f50e80f0b70 msgr2=0x7f50e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.854+0000 7f50f02be700 1 -- 172.21.15.118:0/3525755471 shutdown_connections 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50f02be700 1 -- 172.21.15.118:0/3525755471 wait complete. 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50f02be700 1 Processor -- start 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50f02be700 1 -- start start 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50f02be700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50f02be700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8106cd0 con 0x7f50e80f5410 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50ee05a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50ee05a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54000/0 (socket says 172.21.15.118:54000) 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.855+0000 7f50ee05a700 1 -- 172.21.15.118:0/461863389 learned_addr learned my addr 172.21.15.118:0/461863389 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50ee05a700 1 -- 172.21.15.118:0/461863389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc005040 con 0x7f50e80f5410 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50ee05a700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f50dc005010 tx=0x7f50dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:46.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50dc009320 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e8106ed0 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50dc013380 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50dc014e50 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.856+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e8107360 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50dc014420 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e8045ad0 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50daffd700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 0x7f50d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50dc040e60 con 0x7f50e80f5410 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50ed859700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 0x7f50d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.857+0000 7f50ed859700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 0x7f50d40323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:46.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:46.861+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50dc0095f0 con 0x7f50e80f5410 2024-02-23T00:27:47.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.065+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50e8108210 con 0x7f50e80f5410 2024-02-23T00:27:47.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.066+0000 7f50daffd700 1 -- 172.21.15.118:0/461863389 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50dc01db70 con 0x7f50e80f5410 2024-02-23T00:27:47.067 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:47.067 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 msgr2=0x7f50d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 0x7f50d40323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).stop 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 msgr2=0x7f50e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f50dc005010 tx=0x7f50dc00b040).stop 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 shutdown_connections 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f50d402ff10 0x7f50d40323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 --2- 172.21.15.118:0/461863389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50e80f5410 0x7f50e81067e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 >> 172.21.15.118:0/461863389 conn(0x7f50e80f0b70 msgr2=0x7f50e80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 shutdown_connections 2024-02-23T00:27:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:47.067+0000 7f50f02be700 1 -- 172.21.15.118:0/461863389 wait complete. 2024-02-23T00:27:47.070 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:47.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:47.066415+0000 mon.smithi090 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.118:0/461863389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:48.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:48 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:47.151710+0000 mgr.smithi090.pxxmlz (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:48.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:48.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:49.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:49.151990+0000 mgr.smithi090.pxxmlz (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:50.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- 172.21.15.118:0/2862907144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 msgr2=0x7feb740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:50.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 --2- 172.21.15.118:0/2862907144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7feb64009fb0 tx=0x7feb64009b30).stop 2024-02-23T00:27:50.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- 172.21.15.118:0/2862907144 shutdown_connections 2024-02-23T00:27:50.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 --2- 172.21.15.118:0/2862907144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- 172.21.15.118:0/2862907144 >> 172.21.15.118:0/2862907144 conn(0x7feb740f0b70 msgr2=0x7feb740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- 172.21.15.118:0/2862907144 shutdown_connections 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- 172.21.15.118:0/2862907144 wait complete. 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 Processor -- start 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- start start 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.783+0000 7feb7b2df700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb740fb960 con 0x7feb740f5410 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7907b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7907b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54028/0 (socket says 172.21.15.118:54028) 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7907b700 1 -- 172.21.15.118:0/3580596408 learned_addr learned my addr 172.21.15.118:0/3580596408 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7907b700 1 -- 172.21.15.118:0/3580596408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64005040 con 0x7feb740f5410 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7907b700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7feb640089f0 tx=0x7feb6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:50.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb64009320 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb740fbb00 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb64013380 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6401ed80 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.784+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb740fbf90 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.785+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb6401e420 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.785+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb74059100 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.785+0000 7feb69ffb700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.785+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb64040ff0 con 0x7feb740f5410 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.786+0000 7feb7887a700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.786+0000 7feb7887a700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 0x7feb60032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feb700096a0 tx=0x7feb70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:50.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.790+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb640095f0 con 0x7feb740f5410 2024-02-23T00:27:50.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.994+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb74045ad0 con 0x7feb740f5410 2024-02-23T00:27:50.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.995+0000 7feb69ffb700 1 -- 172.21.15.118:0/3580596408 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb6401cb10 con 0x7feb740f5410 2024-02-23T00:27:50.998 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:50.998 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:50.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 msgr2=0x7feb60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:50.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 0x7feb60032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feb700096a0 tx=0x7feb70006b40).stop 2024-02-23T00:27:50.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 msgr2=0x7feb740fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:50.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7feb640089f0 tx=0x7feb6400b040).stop 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 shutdown_connections 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7feb6002ff60 0x7feb60032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 --2- 172.21.15.118:0/3580596408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb740f5410 0x7feb740fd4e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 >> 172.21.15.118:0/3580596408 conn(0x7feb740f0b70 msgr2=0x7feb7405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 shutdown_connections 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:50.997+0000 7feb7b2df700 1 -- 172.21.15.118:0/3580596408 wait complete. 2024-02-23T00:27:51.000 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:51.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:50.997093+0000 mon.smithi090 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.118:0/3580596408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:51.152234+0000 mgr.smithi090.pxxmlz (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:52.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:52.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:53.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:53.152527+0000 mgr.smithi090.pxxmlz (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:54.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- 172.21.15.118:0/3440913719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 msgr2=0x7f17d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 --2- 172.21.15.118:0/3440913719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d40f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f17c8009fb0 tx=0x7f17c8009b30).stop 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- 172.21.15.118:0/3440913719 shutdown_connections 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 --2- 172.21.15.118:0/3440913719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d40f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- 172.21.15.118:0/3440913719 >> 172.21.15.118:0/3440913719 conn(0x7f17d40f1c70 msgr2=0x7f17d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- 172.21.15.118:0/3440913719 shutdown_connections 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- 172.21.15.118:0/3440913719 wait complete. 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 Processor -- start 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.707+0000 7f17dc07c700 1 -- start start 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17dc07c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:54.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17dc07c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4106cd0 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17d9e18700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17d9e18700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55152/0 (socket says 172.21.15.118:55152) 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17d9e18700 1 -- 172.21.15.118:0/1624943093 learned_addr learned my addr 172.21.15.118:0/1624943093 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17d9e18700 1 -- 172.21.15.118:0/1624943093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8005040 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17d9e18700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f17c8005010 tx=0x7f17c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17c8009320 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4106ed0 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17c8013380 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17c8014e50 con 0x7f17d40f6510 2024-02-23T00:27:54.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.708+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d4107360 con 0x7f17d40f6510 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.709+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17c80095f0 con 0x7f17d40f6510 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.709+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d4045ad0 con 0x7f17d40f6510 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.709+0000 7f17c67fc700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 0x7f17c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.709+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17c8040fa0 con 0x7f17d40f6510 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.710+0000 7f17d9617700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 0x7f17c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:54.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.710+0000 7f17d9617700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 0x7f17c00323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f17d00096a0 tx=0x7f17d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:54.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.714+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17c8014420 con 0x7f17d40f6510 2024-02-23T00:27:54.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.919+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17d4108210 con 0x7f17d40f6510 2024-02-23T00:27:54.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.919+0000 7f17c67fc700 1 -- 172.21.15.118:0/1624943093 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17c8010230 con 0x7f17d40f6510 2024-02-23T00:27:54.922 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:54.922 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 msgr2=0x7f17c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 0x7f17c00323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f17d00096a0 tx=0x7f17d0006b40).stop 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 msgr2=0x7f17d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f17c8005010 tx=0x7f17c800b040).stop 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 shutdown_connections 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f17c002ff10 0x7f17c00323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 --2- 172.21.15.118:0/1624943093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d40f6510 0x7f17d41067e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 >> 172.21.15.118:0/1624943093 conn(0x7f17d40f1c70 msgr2=0x7f17d40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 shutdown_connections 2024-02-23T00:27:54.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:54.921+0000 7f17dc07c700 1 -- 172.21.15.118:0/1624943093 wait complete. 2024-02-23T00:27:54.925 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:55.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:54.921341+0000 mon.smithi090 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.118:0/1624943093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:27:56.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:55.152872+0000 mgr.smithi090.pxxmlz (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:56.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:27:56.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:27:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:57.153161+0000 mgr.smithi090.pxxmlz (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/4076977717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 msgr2=0x7f3f080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/4076977717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f080f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3ef8009fb0 tx=0x7f3ef8009b30).stop 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/4076977717 shutdown_connections 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/4076977717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f080f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/4076977717 >> 172.21.15.118:0/4076977717 conn(0x7f3f080f0b70 msgr2=0x7f3f080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:58.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/4076977717 shutdown_connections 2024-02-23T00:27:58.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/4076977717 wait complete. 2024-02-23T00:27:58.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.564+0000 7f3f0e8f3700 1 Processor -- start 2024-02-23T00:27:58.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f0e8f3700 1 -- start start 2024-02-23T00:27:58.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f0e8f3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:58.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f0e8f3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f08105bd0 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f07fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f07fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55164/0 (socket says 172.21.15.118:55164) 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f07fff700 1 -- 172.21.15.118:0/2462130260 learned_addr learned my addr 172.21.15.118:0/2462130260 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f07fff700 1 -- 172.21.15.118:0/2462130260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef8005040 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.565+0000 7f3f07fff700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3ef8005010 tx=0x7f3ef800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ef8009320 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ef8013380 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f08105dd0 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ef8014e50 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f08106260 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.566+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ef80263a0 con 0x7f3f080f5410 2024-02-23T00:27:58.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.567+0000 7f3f04ff9700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 0x7f3ef40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:58.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.567+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ef8040eb0 con 0x7f3f080f5410 2024-02-23T00:27:58.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.567+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f08045ad0 con 0x7f3f080f5410 2024-02-23T00:27:58.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.567+0000 7f3f077fe700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 0x7f3ef40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:58.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.567+0000 7f3f077fe700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 0x7f3ef40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3ef00096a0 tx=0x7f3ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:58.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.571+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ef8012920 con 0x7f3f080f5410 2024-02-23T00:27:58.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.779+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f08106f80 con 0x7f3f080f5410 2024-02-23T00:27:58.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.779+0000 7f3f04ff9700 1 -- 172.21.15.118:0/2462130260 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ef8012330 con 0x7f3f080f5410 2024-02-23T00:27:58.781 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:27:58.781 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:27:58.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 msgr2=0x7f3ef40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:58.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 0x7f3ef40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3ef00096a0 tx=0x7f3ef0006b40).stop 2024-02-23T00:27:58.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 msgr2=0x7f3f081056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:58.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f3ef8005010 tx=0x7f3ef800b040).stop 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 shutdown_connections 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3ef402ff10 0x7f3ef40323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 --2- 172.21.15.118:0/2462130260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080f5410 0x7f3f081056e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 >> 172.21.15.118:0/2462130260 conn(0x7f3f080f0b70 msgr2=0x7f3f080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 shutdown_connections 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:27:58.781+0000 7f3f0e8f3700 1 -- 172.21.15.118:0/2462130260 wait complete. 2024-02-23T00:27:58.783 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:27:59.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:27:58.780267+0000 mon.smithi090 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.118:0/2462130260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:00.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:27:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:27:59.153461+0000 mgr.smithi090.pxxmlz (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:00.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:00.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:01.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:01.153814+0000 mgr.smithi090.pxxmlz (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 -- 172.21.15.118:0/2597458320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 msgr2=0x7f6c2c069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 --2- 172.21.15.118:0/2597458320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c069e40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6c1c009fb0 tx=0x7f6c1c009b30).stop 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 -- 172.21.15.118:0/2597458320 shutdown_connections 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 --2- 172.21.15.118:0/2597458320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c069e40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 -- 172.21.15.118:0/2597458320 >> 172.21.15.118:0/2597458320 conn(0x7f6c2c0f1c70 msgr2=0x7f6c2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 -- 172.21.15.118:0/2597458320 shutdown_connections 2024-02-23T00:28:02.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 -- 172.21.15.118:0/2597458320 wait complete. 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.463+0000 7f6c326c1700 1 Processor -- start 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c326c1700 1 -- start start 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c326c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c326c1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c108e90 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55184/0 (socket says 172.21.15.118:55184) 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c2bfff700 1 -- 172.21.15.118:0/4113243831 learned_addr learned my addr 172.21.15.118:0/4113243831 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.464+0000 7f6c2bfff700 1 -- 172.21.15.118:0/4113243831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c005040 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c2bfff700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6c1c0089f0 tx=0x7f6c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c1c009320 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c109090 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c1c013380 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c109520 con 0x7f6c2c06b940 2024-02-23T00:28:02.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.465+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c1c014e20 con 0x7f6c2c06b940 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c1c0095f0 con 0x7f6c2c06b940 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c045ad0 con 0x7f6c2c06b940 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c28ff9700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 0x7f6c180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c1c040ea0 con 0x7f6c2c06b940 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c2b7fe700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 0x7f6c180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.466+0000 7f6c2b7fe700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 0x7f6c180323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6c140096a0 tx=0x7f6c14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:02.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.470+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c1c014420 con 0x7f6c2c06b940 2024-02-23T00:28:02.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.678+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c2c10a370 con 0x7f6c2c06b940 2024-02-23T00:28:02.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.679+0000 7f6c28ff9700 1 -- 172.21.15.118:0/4113243831 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c1c010230 con 0x7f6c2c06b940 2024-02-23T00:28:02.681 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 msgr2=0x7f6c180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 0x7f6c180323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6c140096a0 tx=0x7f6c14006b40).stop 2024-02-23T00:28:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 msgr2=0x7f6c2c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6c1c0089f0 tx=0x7f6c1c00b040).stop 2024-02-23T00:28:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 shutdown_connections 2024-02-23T00:28:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c1802ff10 0x7f6c180323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 --2- 172.21.15.118:0/4113243831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c2c06b940 0x7f6c2c1089a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 >> 172.21.15.118:0/4113243831 conn(0x7f6c2c0f1c70 msgr2=0x7f6c2c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.680+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 shutdown_connections 2024-02-23T00:28:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:02.681+0000 7f6c326c1700 1 -- 172.21.15.118:0/4113243831 wait complete. 2024-02-23T00:28:02.687 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:03.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:02.679775+0000 mon.smithi090 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.118:0/4113243831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:04.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:03.154093+0000 mgr.smithi090.pxxmlz (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:04.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:04.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:05.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:05.154331+0000 mgr.smithi090.pxxmlz (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 -- 172.21.15.118:0/2675038443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 msgr2=0x7fd67c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 --2- 172.21.15.118:0/2675038443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c0f68a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fd678009fb0 tx=0x7fd678009b30).stop 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 -- 172.21.15.118:0/2675038443 shutdown_connections 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 --2- 172.21.15.118:0/2675038443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c0f68a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 -- 172.21.15.118:0/2675038443 >> 172.21.15.118:0/2675038443 conn(0x7fd67c0f1c00 msgr2=0x7fd67c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 -- 172.21.15.118:0/2675038443 shutdown_connections 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 -- 172.21.15.118:0/2675038443 wait complete. 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.604+0000 7fd68445b700 1 Processor -- start 2024-02-23T00:28:06.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd68445b700 1 -- start start 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd68445b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd68445b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c106ce0 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd6821f7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd6821f7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42964/0 (socket says 172.21.15.118:42964) 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd6821f7700 1 -- 172.21.15.118:0/61505053 learned_addr learned my addr 172.21.15.118:0/61505053 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd6821f7700 1 -- 172.21.15.118:0/61505053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678005040 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd6821f7700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fd678005010 tx=0x7fd67800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd678009320 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c106ee0 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd678013380 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.605+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd678014e50 con 0x7fd67c0f6480 2024-02-23T00:28:06.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.606+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c107370 con 0x7fd67c0f6480 2024-02-23T00:28:06.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.606+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6780095f0 con 0x7fd67c0f6480 2024-02-23T00:28:06.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.606+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c045ad0 con 0x7fd67c0f6480 2024-02-23T00:28:06.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.606+0000 7fd672ffd700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 0x7fd6680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.606+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd678040fa0 con 0x7fd67c0f6480 2024-02-23T00:28:06.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.607+0000 7fd6819f6700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 0x7fd6680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.607+0000 7fd6819f6700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 0x7fd6680323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd66c0096a0 tx=0x7fd66c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:06.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.611+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd678014420 con 0x7fd67c0f6480 2024-02-23T00:28:06.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.816+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd67c0fc090 con 0x7fd67c0f6480 2024-02-23T00:28:06.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.817+0000 7fd672ffd700 1 -- 172.21.15.118:0/61505053 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd678010230 con 0x7fd67c0f6480 2024-02-23T00:28:06.818 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:06.819 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 msgr2=0x7fd6680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 0x7fd6680323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd66c0096a0 tx=0x7fd66c006b40).stop 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 msgr2=0x7fd67c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fd678005010 tx=0x7fd67800b040).stop 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 shutdown_connections 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd66802ff10 0x7fd6680323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 --2- 172.21.15.118:0/61505053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd67c0f6480 0x7fd67c1067f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.818+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 >> 172.21.15.118:0/61505053 conn(0x7fd67c0f1c00 msgr2=0x7fd67c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.819+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 shutdown_connections 2024-02-23T00:28:06.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:06.819+0000 7fd68445b700 1 -- 172.21.15.118:0/61505053 wait complete. 2024-02-23T00:28:06.821 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:07.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:06.817892+0000 mon.smithi090 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.118:0/61505053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:08.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:07.154558+0000 mgr.smithi090.pxxmlz (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:08.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:08.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:09.154837+0000 mgr.smithi090.pxxmlz (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- 172.21.15.118:0/2280198474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 msgr2=0x7ff3340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 --2- 172.21.15.118:0/2280198474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3340f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff324009fb0 tx=0x7ff324009b30).stop 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- 172.21.15.118:0/2280198474 shutdown_connections 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 --2- 172.21.15.118:0/2280198474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3340f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- 172.21.15.118:0/2280198474 >> 172.21.15.118:0/2280198474 conn(0x7ff3340f0b70 msgr2=0x7ff3340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- 172.21.15.118:0/2280198474 shutdown_connections 2024-02-23T00:28:10.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- 172.21.15.118:0/2280198474 wait complete. 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 Processor -- start 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- start start 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.414+0000 7ff33b566700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334106cd0 con 0x7ff3340f5410 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff339302700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff339302700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42984/0 (socket says 172.21.15.118:42984) 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff339302700 1 -- 172.21.15.118:0/2751599284 learned_addr learned my addr 172.21.15.118:0/2751599284 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff339302700 1 -- 172.21.15.118:0/2751599284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324005040 con 0x7ff3340f5410 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff339302700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff324005010 tx=0x7ff32400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff324009320 con 0x7ff3340f5410 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334106ed0 con 0x7ff3340f5410 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff324013380 con 0x7ff3340f5410 2024-02-23T00:28:10.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff324014e50 con 0x7ff3340f5410 2024-02-23T00:28:10.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.415+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff334107360 con 0x7ff3340f5410 2024-02-23T00:28:10.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.416+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3240095f0 con 0x7ff3340f5410 2024-02-23T00:28:10.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.416+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334045ad0 con 0x7ff3340f5410 2024-02-23T00:28:10.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.416+0000 7ff329ffb700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 0x7ff3200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:10.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.416+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff324040e90 con 0x7ff3340f5410 2024-02-23T00:28:10.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.417+0000 7ff338b01700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 0x7ff3200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:10.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.417+0000 7ff338b01700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 0x7ff3200323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff3300096a0 tx=0x7ff330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:10.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.421+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff324014420 con 0x7ff3340f5410 2024-02-23T00:28:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.626+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3341081b0 con 0x7ff3340f5410 2024-02-23T00:28:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.626+0000 7ff329ffb700 1 -- 172.21.15.118:0/2751599284 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff324010230 con 0x7ff3340f5410 2024-02-23T00:28:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:10.628 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 msgr2=0x7ff3200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 0x7ff3200323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff3300096a0 tx=0x7ff330006b40).stop 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 msgr2=0x7ff3341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff324005010 tx=0x7ff32400b040).stop 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 shutdown_connections 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff32002ff10 0x7ff3200323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 --2- 172.21.15.118:0/2751599284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3340f5410 0x7ff3341067e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 >> 172.21.15.118:0/2751599284 conn(0x7ff3340f0b70 msgr2=0x7ff3340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:10.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 shutdown_connections 2024-02-23T00:28:10.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:10.628+0000 7ff33b566700 1 -- 172.21.15.118:0/2751599284 wait complete. 2024-02-23T00:28:10.631 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:10.627170+0000 mon.smithi090 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.118:0/2751599284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:11.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:11.155212+0000 mgr.smithi090.pxxmlz (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:12.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:12.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:13.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:13.155546+0000 mgr.smithi090.pxxmlz (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:14.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 -- 172.21.15.118:0/3245509496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 msgr2=0x7fe0000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:14.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 --2- 172.21.15.118:0/3245509496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0000f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fdffc009fb0 tx=0x7fdffc009b30).stop 2024-02-23T00:28:14.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 -- 172.21.15.118:0/3245509496 shutdown_connections 2024-02-23T00:28:14.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 --2- 172.21.15.118:0/3245509496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0000f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:14.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 -- 172.21.15.118:0/3245509496 >> 172.21.15.118:0/3245509496 conn(0x7fe0000f1c70 msgr2=0x7fe0000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 -- 172.21.15.118:0/3245509496 shutdown_connections 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 -- 172.21.15.118:0/3245509496 wait complete. 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.481+0000 7fe0082a6700 1 Processor -- start 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe0082a6700 1 -- start start 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe0082a6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe0082a6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000106cd0 con 0x7fe0000f6510 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe006042700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe006042700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48938/0 (socket says 172.21.15.118:48938) 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe006042700 1 -- 172.21.15.118:0/312037828 learned_addr learned my addr 172.21.15.118:0/312037828 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe006042700 1 -- 172.21.15.118:0/312037828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc005040 con 0x7fe0000f6510 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe006042700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdffc005010 tx=0x7fdffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc009320 con 0x7fe0000f6510 2024-02-23T00:28:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe000106ed0 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdffc013380 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc014e50 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.482+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe000107360 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.483+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe000045ad0 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.483+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdffc0095f0 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.483+0000 7fdff6ffd700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 0x7fdfec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.483+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdffc040e90 con 0x7fe0000f6510 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.484+0000 7fe005841700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 0x7fdfec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.484+0000 7fe005841700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 0x7fdfec0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:14.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.488+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdffc014420 con 0x7fe0000f6510 2024-02-23T00:28:14.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.693+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe000108210 con 0x7fe0000f6510 2024-02-23T00:28:14.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.694+0000 7fdff6ffd700 1 -- 172.21.15.118:0/312037828 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdffc010230 con 0x7fe0000f6510 2024-02-23T00:28:14.695 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:14.695 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.695+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 msgr2=0x7fdfec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.695+0000 7fe0082a6700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 0x7fdfec0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fdff00096a0 tx=0x7fdff0006b40).stop 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.695+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 msgr2=0x7fe0001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.695+0000 7fe0082a6700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdffc005010 tx=0x7fdffc00b040).stop 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.695+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 shutdown_connections 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.696+0000 7fe0082a6700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fdfec02ff10 0x7fdfec0323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:14.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.696+0000 7fe0082a6700 1 --2- 172.21.15.118:0/312037828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0000f6510 0x7fe0001067e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:14.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.696+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 >> 172.21.15.118:0/312037828 conn(0x7fe0000f1c70 msgr2=0x7fe0000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:14.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.696+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 shutdown_connections 2024-02-23T00:28:14.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:14.696+0000 7fe0082a6700 1 -- 172.21.15.118:0/312037828 wait complete. 2024-02-23T00:28:14.698 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:15.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:14.694709+0000 mon.smithi090 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.118:0/312037828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:16.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:15.155814+0000 mgr.smithi090.pxxmlz (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:16.434 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:16.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:17.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:17.156124+0000 mgr.smithi090.pxxmlz (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:18.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 -- 172.21.15.118:0/3798015007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 msgr2=0x7fbe340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/3798015007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe340f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fbe2c009960 tx=0x7fbe2c004d10).stop 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 -- 172.21.15.118:0/3798015007 shutdown_connections 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/3798015007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe340f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 -- 172.21.15.118:0/3798015007 >> 172.21.15.118:0/3798015007 conn(0x7fbe340f0b70 msgr2=0x7fbe340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 -- 172.21.15.118:0/3798015007 shutdown_connections 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 -- 172.21.15.118:0/3798015007 wait complete. 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.370+0000 7fbe39aa9700 1 Processor -- start 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe39aa9700 1 -- start start 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe39aa9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe39aa9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe34060220 con 0x7fbe340f5410 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe337fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe337fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48958/0 (socket says 172.21.15.118:48958) 2024-02-23T00:28:18.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.371+0000 7fbe337fe700 1 -- 172.21.15.118:0/1046537220 learned_addr learned my addr 172.21.15.118:0/1046537220 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe337fe700 1 -- 172.21.15.118:0/1046537220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c009640 con 0x7fbe340f5410 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe337fe700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbe2c004b70 tx=0x7fbe2c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c018070 con 0x7fbe340f5410 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe34060420 con 0x7fbe340f5410 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe2c00b970 con 0x7fbe340f5410 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c01c560 con 0x7fbe340f5410 2024-02-23T00:28:18.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.372+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe34109bb0 con 0x7fbe340f5410 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe2c013630 con 0x7fbe340f5410 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe34059100 con 0x7fbe340f5410 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe1bfff700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe2c041320 con 0x7fbe340f5410 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe32ffd700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:18.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.373+0000 7fbe32ffd700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 0x7fbe1c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:18.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.377+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe2c012970 con 0x7fbe340f5410 2024-02-23T00:28:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.584+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe34045ad0 con 0x7fbe340f5410 2024-02-23T00:28:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.584+0000 7fbe1bfff700 1 -- 172.21.15.118:0/1046537220 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe2c012380 con 0x7fbe340f5410 2024-02-23T00:28:18.586 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:18.586 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 msgr2=0x7fbe1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 0x7fbe1c032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).stop 2024-02-23T00:28:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 msgr2=0x7fbe34109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbe2c004b70 tx=0x7fbe2c004d10).stop 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 shutdown_connections 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 --2- 172.21.15.118:0/1046537220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe340f5410 0x7fbe34109500 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 >> 172.21.15.118:0/1046537220 conn(0x7fbe340f0b70 msgr2=0x7fbe340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 shutdown_connections 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:18.586+0000 7fbe39aa9700 1 -- 172.21.15.118:0/1046537220 wait complete. 2024-02-23T00:28:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:19.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:18.585195+0000 mon.smithi090 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.118:0/1046537220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:20.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:19.156474+0000 mgr.smithi090.pxxmlz (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:20.207 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:20.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:21.156851+0000 mgr.smithi090.pxxmlz (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 -- 172.21.15.118:0/2356858199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 msgr2=0x7ff68c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 --2- 172.21.15.118:0/2356858199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f4be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7ff67c009fb0 tx=0x7ff67c009b30).stop 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 -- 172.21.15.118:0/2356858199 shutdown_connections 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 --2- 172.21.15.118:0/2356858199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f4be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 -- 172.21.15.118:0/2356858199 >> 172.21.15.118:0/2356858199 conn(0x7ff68c0f02e0 msgr2=0x7ff68c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 -- 172.21.15.118:0/2356858199 shutdown_connections 2024-02-23T00:28:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.012+0000 7ff694736700 1 -- 172.21.15.118:0/2356858199 wait complete. 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff694736700 1 Processor -- start 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff694736700 1 -- start start 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff694736700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff694736700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0f57f0 con 0x7ff68c0f47c0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff6924d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff6924d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48984/0 (socket says 172.21.15.118:48984) 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff6924d2700 1 -- 172.21.15.118:0/3568993293 learned_addr learned my addr 172.21.15.118:0/3568993293 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.013+0000 7ff6924d2700 1 -- 172.21.15.118:0/3568993293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c005040 con 0x7ff68c0f47c0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff6924d2700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff67c005010 tx=0x7ff67c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff67c009320 con 0x7ff68c0f47c0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c0f5990 con 0x7ff68c0f47c0 2024-02-23T00:28:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff67c013380 con 0x7ff68c0f47c0 2024-02-23T00:28:22.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff67c014e50 con 0x7ff68c0f47c0 2024-02-23T00:28:22.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.014+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c0f5e20 con 0x7ff68c0f47c0 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff67c0095f0 con 0x7ff68c0f47c0 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c059100 con 0x7ff68c0f47c0 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff682ffd700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 0x7ff6780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff67c040e90 con 0x7ff68c0f47c0 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff691cd1700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 0x7ff6780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.015+0000 7ff691cd1700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 0x7ff6780323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff6880096a0 tx=0x7ff688006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:22.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.019+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff67c014420 con 0x7ff68c0f47c0 2024-02-23T00:28:22.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.223+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff68c0fc090 con 0x7ff68c0f47c0 2024-02-23T00:28:22.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.224+0000 7ff682ffd700 1 -- 172.21.15.118:0/3568993293 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff67c010230 con 0x7ff68c0f47c0 2024-02-23T00:28:22.230 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:22.230 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 msgr2=0x7ff6780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 0x7ff6780323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff6880096a0 tx=0x7ff688006b40).stop 2024-02-23T00:28:22.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 msgr2=0x7ff68c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7ff67c005010 tx=0x7ff67c00b040).stop 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 shutdown_connections 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff67802ff10 0x7ff6780323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.225+0000 7ff694736700 1 --2- 172.21.15.118:0/3568993293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff68c0f47c0 0x7ff68c0f7370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.226+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 >> 172.21.15.118:0/3568993293 conn(0x7ff68c0f02e0 msgr2=0x7ff68c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.226+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 shutdown_connections 2024-02-23T00:28:22.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:22.226+0000 7ff694736700 1 -- 172.21.15.118:0/3568993293 wait complete. 2024-02-23T00:28:22.233 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:22.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:22.224907+0000 mon.smithi090 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.118:0/3568993293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:23.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:23.157169+0000 mgr.smithi090.pxxmlz (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:23.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:23.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:25.157415+0000 mgr.smithi090.pxxmlz (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.990+0000 7f5280de0700 1 -- 172.21.15.118:0/3835557472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 msgr2=0x7f527c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.990+0000 7f5280de0700 1 --2- 172.21.15.118:0/3835557472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f5270009fb0 tx=0x7f5270009b30).stop 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 -- 172.21.15.118:0/3835557472 shutdown_connections 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 --2- 172.21.15.118:0/3835557472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 -- 172.21.15.118:0/3835557472 >> 172.21.15.118:0/3835557472 conn(0x7f527c0f0b70 msgr2=0x7f527c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 -- 172.21.15.118:0/3835557472 shutdown_connections 2024-02-23T00:28:25.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 -- 172.21.15.118:0/3835557472 wait complete. 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 Processor -- start 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 -- start start 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.991+0000 7f5280de0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f5280de0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c060240 con 0x7f527c0f5410 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f527a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f527a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59848/0 (socket says 172.21.15.118:59848) 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f527a59c700 1 -- 172.21.15.118:0/823097335 learned_addr learned my addr 172.21.15.118:0/823097335 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f527a59c700 1 -- 172.21.15.118:0/823097335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5270005040 con 0x7f527c0f5410 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f527a59c700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5270005010 tx=0x7f527000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5270009320 con 0x7f527c0f5410 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c060440 con 0x7f527c0f5410 2024-02-23T00:28:25.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5270013380 con 0x7f527c0f5410 2024-02-23T00:28:25.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5270014e50 con 0x7f527c0f5410 2024-02-23T00:28:25.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.992+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c109b90 con 0x7f527c0f5410 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.993+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52700095f0 con 0x7f527c0f5410 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.993+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c045ad0 con 0x7f527c0f5410 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.993+0000 7f526affd700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 0x7f52640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.993+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5270040fa0 con 0x7f527c0f5410 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.994+0000 7f5279d9b700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 0x7f52640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:25.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.994+0000 7f5279d9b700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 0x7f52640323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f526c0096a0 tx=0x7f526c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:25.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:25.998+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5270014420 con 0x7f527c0f5410 2024-02-23T00:28:26.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.204+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f527c10a3b0 con 0x7f527c0f5410 2024-02-23T00:28:26.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.205+0000 7f526affd700 1 -- 172.21.15.118:0/823097335 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5270010230 con 0x7f527c0f5410 2024-02-23T00:28:26.206 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:26.207 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 msgr2=0x7f52640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 0x7f52640323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f526c0096a0 tx=0x7f526c006b40).stop 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 msgr2=0x7f527c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5270005010 tx=0x7f527000b040).stop 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 shutdown_connections 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f526402ff10 0x7f52640323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 --2- 172.21.15.118:0/823097335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f527c0f5410 0x7f527c1094e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 >> 172.21.15.118:0/823097335 conn(0x7f527c0f0b70 msgr2=0x7f527c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:26.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 shutdown_connections 2024-02-23T00:28:26.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:26.207+0000 7f5280de0700 1 -- 172.21.15.118:0/823097335 wait complete. 2024-02-23T00:28:26.209 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:26.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:25.247130+0000 mon.smithi090 (mon.0) 190 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:28:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:25.253998+0000 mon.smithi090 (mon.0) 191 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:28:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:26.205905+0000 mon.smithi090 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.118:0/823097335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:27.157706+0000 mgr.smithi090.pxxmlz (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:27.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:27.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:29.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:29.158033+0000 mgr.smithi090.pxxmlz (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/3967444791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 msgr2=0x7f0bc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/3967444791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc00f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f0bb4009fb0 tx=0x7f0bb4009b30).stop 2024-02-23T00:28:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/3967444791 shutdown_connections 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/3967444791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc00f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/3967444791 >> 172.21.15.118:0/3967444791 conn(0x7f0bc00f1c70 msgr2=0x7f0bc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/3967444791 shutdown_connections 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.111+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/3967444791 wait complete. 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc6ac0700 1 Processor -- start 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc6ac0700 1 -- start start 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc6ac0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc6ac0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0069e20 con 0x7f0bc00f6510 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc485c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc485c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59862/0 (socket says 172.21.15.118:59862) 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.112+0000 7f0bc485c700 1 -- 172.21.15.118:0/2803161977 learned_addr learned my addr 172.21.15.118:0/2803161977 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bc485c700 1 -- 172.21.15.118:0/2803161977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb4005040 con 0x7f0bc00f6510 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bc485c700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f0bb4005010 tx=0x7f0bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:30.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb4009320 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bb4013380 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc0069fc0 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bb4014e50 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.113+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc006a450 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.114+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bb4014420 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.114+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0059100 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.114+0000 7f0bbd7fa700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.114+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bb4040f70 con 0x7f0bc00f6510 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.114+0000 7f0bbffff700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.115+0000 7f0bbffff700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 0x7f0bb00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f0bac0096a0 tx=0x7f0bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:30.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.119+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bb40095f0 con 0x7f0bc00f6510 2024-02-23T00:28:30.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.325+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bc00fc0a0 con 0x7f0bc00f6510 2024-02-23T00:28:30.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.327+0000 7f0bbd7fa700 1 -- 172.21.15.118:0/2803161977 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bb401db70 con 0x7f0bc00f6510 2024-02-23T00:28:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:30.328 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 msgr2=0x7f0bb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 0x7f0bb00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f0bac0096a0 tx=0x7f0bac006b40).stop 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 msgr2=0x7f0bc006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f0bb4005010 tx=0x7f0bb400b040).stop 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 shutdown_connections 2024-02-23T00:28:30.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f0bb002ff10 0x7f0bb00323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 --2- 172.21.15.118:0/2803161977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bc00f6510 0x7f0bc006b9a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 >> 172.21.15.118:0/2803161977 conn(0x7f0bc00f1c70 msgr2=0x7f0bc00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:30.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 shutdown_connections 2024-02-23T00:28:30.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:30.328+0000 7f0bc6ac0700 1 -- 172.21.15.118:0/2803161977 wait complete. 2024-02-23T00:28:30.330 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:30.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:30.326994+0000 mon.smithi090 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.118:0/2803161977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:31.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:31.158410+0000 mgr.smithi090.pxxmlz (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:32.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:32.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:33.158748+0000 mgr.smithi090.pxxmlz (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- 172.21.15.118:0/918997970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 msgr2=0x7f075c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 --2- 172.21.15.118:0/918997970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c0f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f074c009fb0 tx=0x7f074c009b30).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- 172.21.15.118:0/918997970 shutdown_connections 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 --2- 172.21.15.118:0/918997970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c0f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- 172.21.15.118:0/918997970 >> 172.21.15.118:0/918997970 conn(0x7f075c0f0b70 msgr2=0x7f075c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- 172.21.15.118:0/918997970 shutdown_connections 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- 172.21.15.118:0/918997970 wait complete. 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 Processor -- start 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- start start 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.235+0000 7f076390d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c105bd0 con 0x7f075c0f5410 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f07616a9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f07616a9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59880/0 (socket says 172.21.15.118:59880) 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f07616a9700 1 -- 172.21.15.118:0/1947280053 learned_addr learned my addr 172.21.15.118:0/1947280053 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f07616a9700 1 -- 172.21.15.118:0/1947280053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c005040 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f07616a9700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f074c005010 tx=0x7f074c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074c009320 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c105dd0 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f074c013380 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.236+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f074c014e50 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.237+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c106260 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.237+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f074c014420 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.238+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c045ad0 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.238+0000 7f0751ffb700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 0x7f07480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.238+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f074c040f70 con 0x7f075c0f5410 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.238+0000 7f0760ea8700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 0x7f07480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.238+0000 7f0760ea8700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 0x7f07480323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f07580096a0 tx=0x7f0758006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.243+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f074c0095f0 con 0x7f075c0f5410 2024-02-23T00:28:34.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.452+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f075c107090 con 0x7f075c0f5410 2024-02-23T00:28:34.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.453+0000 7f0751ffb700 1 -- 172.21.15.118:0/1947280053 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f074c01db70 con 0x7f075c0f5410 2024-02-23T00:28:34.455 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:34.455 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.454+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 msgr2=0x7f07480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.454+0000 7f076390d700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 0x7f07480323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f07580096a0 tx=0x7f0758006b40).stop 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.454+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 msgr2=0x7f075c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.454+0000 7f076390d700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f074c005010 tx=0x7f074c00b040).stop 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 shutdown_connections 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f074802ff10 0x7f07480323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 --2- 172.21.15.118:0/1947280053 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0f5410 0x7f075c1056e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 >> 172.21.15.118:0/1947280053 conn(0x7f075c0f0b70 msgr2=0x7f075c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 shutdown_connections 2024-02-23T00:28:34.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:34.455+0000 7f076390d700 1 -- 172.21.15.118:0/1947280053 wait complete. 2024-02-23T00:28:34.458 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:34.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:34.454807+0000 mon.smithi090 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.118:0/1947280053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:28:34.697670+0000 mgr.smithi090.pxxmlz (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi090 ... 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:24:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:24:50Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:25:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:26:42Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:27:19Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:27:56Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:28:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('uid/gid not found') 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: uid/gid not found 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: yield (conn, connr) 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: code, '\n'.join(err))) 2024-02-23T00:28:35.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi090 ... 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:24:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:24:50Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:25:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:26:42Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:27:19Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:27:56Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:28:35.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('uid/gid not found') 2024-02-23T00:28:35.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: uid/gid not found 2024-02-23T00:28:35.755 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:34.699361+0000 mon.smithi090 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-23T00:28:35.755 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:34.704737+0000 mon.smithi090 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-23T00:28:35.755 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:34.705519+0000 mon.smithi090 (mon.0) 197 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:35.755 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:28:34.706096+0000 mgr.smithi090.pxxmlz (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi090 on smithi090 2024-02-23T00:28:35.755 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:35.159008+0000 mgr.smithi090.pxxmlz (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:36.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:36.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:36.935729+0000 mon.smithi090 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:37.000474+0000 mon.smithi090 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:37.002754+0000 mon.smithi090 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:37.003148+0000 mon.smithi090 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:37.003565+0000 mgr.smithi090.pxxmlz (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:37.005842+0000 mon.smithi090 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:28:37.009093+0000 mgr.smithi090.pxxmlz (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi090 on smithi090 2024-02-23T00:28:38.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:37.159238+0000 mgr.smithi090.pxxmlz (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 -- 172.21.15.118:0/2551481712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 msgr2=0x7f1c880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/2551481712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c880f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1c7c009fb0 tx=0x7f1c7c009b30).stop 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 -- 172.21.15.118:0/2551481712 shutdown_connections 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/2551481712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c880f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 -- 172.21.15.118:0/2551481712 >> 172.21.15.118:0/2551481712 conn(0x7f1c880f0b70 msgr2=0x7f1c880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 -- 172.21.15.118:0/2551481712 shutdown_connections 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.262+0000 7f1c8f15c700 1 -- 172.21.15.118:0/2551481712 wait complete. 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.263+0000 7f1c8f15c700 1 Processor -- start 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.263+0000 7f1c8f15c700 1 -- start start 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.263+0000 7f1c8f15c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.263+0000 7f1c8f15c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c88106cd0 con 0x7f1c880f5410 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8cef8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8cef8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33160/0 (socket says 172.21.15.118:33160) 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8cef8700 1 -- 172.21.15.118:0/3728923140 learned_addr learned my addr 172.21.15.118:0/3728923140 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8cef8700 1 -- 172.21.15.118:0/3728923140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c005040 con 0x7f1c880f5410 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8cef8700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1c7c005010 tx=0x7f1c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c7c009320 con 0x7f1c880f5410 2024-02-23T00:28:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c88106ed0 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c7c013380 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c7c014e50 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.264+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c88107360 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.265+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c7c0095f0 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.265+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88045ad0 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.265+0000 7f1c857fa700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 0x7f1c780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.265+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c7c040fa0 con 0x7f1c880f5410 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.265+0000 7f1c87fff700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 0x7f1c780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.266+0000 7f1c87fff700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 0x7f1c780323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1c740096a0 tx=0x7f1c74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:38.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.270+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c7c014420 con 0x7f1c880f5410 2024-02-23T00:28:38.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.477+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c88108210 con 0x7f1c880f5410 2024-02-23T00:28:38.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.478+0000 7f1c857fa700 1 -- 172.21.15.118:0/3728923140 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c7c010230 con 0x7f1c880f5410 2024-02-23T00:28:38.480 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:38.480 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:38.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 msgr2=0x7f1c780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 0x7f1c780323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1c740096a0 tx=0x7f1c74006b40).stop 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 msgr2=0x7f1c881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1c7c005010 tx=0x7f1c7c00b040).stop 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 shutdown_connections 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1c7802ff10 0x7f1c780323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 --2- 172.21.15.118:0/3728923140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c880f5410 0x7f1c881067e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 >> 172.21.15.118:0/3728923140 conn(0x7f1c880f0b70 msgr2=0x7f1c880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 shutdown_connections 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:38.479+0000 7f1c8f15c700 1 -- 172.21.15.118:0/3728923140 wait complete. 2024-02-23T00:28:38.482 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:39.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:38.479502+0000 mon.smithi090 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.118:0/3728923140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:40.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:40.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:40.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:39.159554+0000 mgr.smithi090.pxxmlz (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:41.159924+0000 mgr.smithi090.pxxmlz (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:42.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 -- 172.21.15.118:0/820767848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 msgr2=0x7f08800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:42.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 --2- 172.21.15.118:0/820767848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08800f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f0870009fb0 tx=0x7f0870009b30).stop 2024-02-23T00:28:42.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 -- 172.21.15.118:0/820767848 shutdown_connections 2024-02-23T00:28:42.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 --2- 172.21.15.118:0/820767848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08800f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 -- 172.21.15.118:0/820767848 >> 172.21.15.118:0/820767848 conn(0x7f08800f0b70 msgr2=0x7f08800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 -- 172.21.15.118:0/820767848 shutdown_connections 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.268+0000 7f0886534700 1 -- 172.21.15.118:0/820767848 wait complete. 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f0886534700 1 Processor -- start 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f0886534700 1 -- start start 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f0886534700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f0886534700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880106ca0 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f087ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f087ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33182/0 (socket says 172.21.15.118:33182) 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.269+0000 7f087ffff700 1 -- 172.21.15.118:0/3654441917 learned_addr learned my addr 172.21.15.118:0/3654441917 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f087ffff700 1 -- 172.21.15.118:0/3654441917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0870005040 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f087ffff700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0870005010 tx=0x7f087000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0870009320 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0880106ea0 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0870013380 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0870014e50 con 0x7f08800f5410 2024-02-23T00:28:42.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.270+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0880107330 con 0x7f08800f5410 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.271+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0870014420 con 0x7f08800f5410 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.271+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880045ad0 con 0x7f08800f5410 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.271+0000 7f087cff9700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 0x7f086c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.271+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0870040f40 con 0x7f08800f5410 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.271+0000 7f087f7fe700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 0x7f086c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:42.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.272+0000 7f087f7fe700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 0x7f086c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08680096a0 tx=0x7f0868006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:42.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.276+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08700095f0 con 0x7f08800f5410 2024-02-23T00:28:42.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.483+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08801081e0 con 0x7f08800f5410 2024-02-23T00:28:42.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.484+0000 7f087cff9700 1 -- 172.21.15.118:0/3654441917 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f087001db70 con 0x7f08800f5410 2024-02-23T00:28:42.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:42.485 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.485+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 msgr2=0x7f086c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.485+0000 7f0886534700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 0x7f086c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08680096a0 tx=0x7f0868006b40).stop 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.485+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 msgr2=0x7f08801067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.485+0000 7f0886534700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0870005010 tx=0x7f087000b040).stop 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 shutdown_connections 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f086c02ff10 0x7f086c0323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 --2- 172.21.15.118:0/3654441917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08800f5410 0x7f08801067b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:42.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 >> 172.21.15.118:0/3654441917 conn(0x7f08800f0b70 msgr2=0x7f088005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:42.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 shutdown_connections 2024-02-23T00:28:42.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:42.486+0000 7f0886534700 1 -- 172.21.15.118:0/3654441917 wait complete. 2024-02-23T00:28:42.488 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:43.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:42.484726+0000 mon.smithi090 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.118:0/3654441917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:44.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:43.160254+0000 mgr.smithi090.pxxmlz (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:44.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:44.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:45.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:45.160502+0000 mgr.smithi090.pxxmlz (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 -- 172.21.15.118:0/1541418586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 msgr2=0x7f58ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1541418586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f58dc009fb0 tx=0x7f58dc009b30).stop 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 -- 172.21.15.118:0/1541418586 shutdown_connections 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1541418586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 -- 172.21.15.118:0/1541418586 >> 172.21.15.118:0/1541418586 conn(0x7f58ec0f0b70 msgr2=0x7f58ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 -- 172.21.15.118:0/1541418586 shutdown_connections 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 -- 172.21.15.118:0/1541418586 wait complete. 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.085+0000 7f58f40b4700 1 Processor -- start 2024-02-23T00:28:46.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f40b4700 1 -- start start 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f40b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f40b4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0fb960 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f1e50700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f1e50700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60908/0 (socket says 172.21.15.118:60908) 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f1e50700 1 -- 172.21.15.118:0/1623259937 learned_addr learned my addr 172.21.15.118:0/1623259937 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f1e50700 1 -- 172.21.15.118:0/1623259937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc005040 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f1e50700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f58dc0089f0 tx=0x7f58dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58dc009320 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec0fbb00 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58dc013380 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58dc01ed80 con 0x7f58ec0f5410 2024-02-23T00:28:46.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.086+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec0fbf90 con 0x7f58ec0f5410 2024-02-23T00:28:46.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.087+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58dc01e420 con 0x7f58ec0f5410 2024-02-23T00:28:46.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.087+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec059100 con 0x7f58ec0f5410 2024-02-23T00:28:46.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.087+0000 7f58e27fc700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 0x7f58d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:46.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.087+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58dc0410d0 con 0x7f58ec0f5410 2024-02-23T00:28:46.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.088+0000 7f58f164f700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 0x7f58d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:46.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.088+0000 7f58f164f700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 0x7f58d8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f58e80096a0 tx=0x7f58e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:46.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.095+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58dc0095f0 con 0x7f58ec0f5410 2024-02-23T00:28:46.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.302+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58ec0fafa0 con 0x7f58ec0f5410 2024-02-23T00:28:46.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.303+0000 7f58e27fc700 1 -- 172.21.15.118:0/1623259937 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58dc02c2d0 con 0x7f58ec0f5410 2024-02-23T00:28:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:46.305 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 msgr2=0x7f58d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 0x7f58d8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f58e80096a0 tx=0x7f58e8006b40).stop 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 msgr2=0x7f58ec0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f58dc0089f0 tx=0x7f58dc00b040).stop 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 shutdown_connections 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f58d802ff60 0x7f58d8032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 --2- 172.21.15.118:0/1623259937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec0f5410 0x7f58ec0fd4e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:46.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 >> 172.21.15.118:0/1623259937 conn(0x7f58ec0f0b70 msgr2=0x7f58ec05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:46.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 shutdown_connections 2024-02-23T00:28:46.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:46.305+0000 7f58f40b4700 1 -- 172.21.15.118:0/1623259937 wait complete. 2024-02-23T00:28:46.307 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:46.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:46.304010+0000 mon.smithi090 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.118:0/1623259937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:47.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:47.160724+0000 mgr.smithi090.pxxmlz (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:47.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:47.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:49.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:49.160966+0000 mgr.smithi090.pxxmlz (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:49.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/3806574564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 msgr2=0x7fa3a00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:49.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/3806574564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a00f66c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa390009fb0 tx=0x7fa390009b30).stop 2024-02-23T00:28:49.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/3806574564 shutdown_connections 2024-02-23T00:28:49.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/3806574564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a00f66c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:49.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/3806574564 >> 172.21.15.118:0/3806574564 conn(0x7fa3a00f1aa0 msgr2=0x7fa3a00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/3806574564 shutdown_connections 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/3806574564 wait complete. 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.984+0000 7fa3a4f0e700 1 Processor -- start 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa3a4f0e700 1 -- start start 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa3a4f0e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa3a4f0e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0106ac0 con 0x7fa3a00f62a0 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa39f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa39f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60922/0 (socket says 172.21.15.118:60922) 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa39f7fe700 1 -- 172.21.15.118:0/1291498548 learned_addr learned my addr 172.21.15.118:0/1291498548 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.985+0000 7fa39f7fe700 1 -- 172.21.15.118:0/1291498548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa390005040 con 0x7fa3a00f62a0 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa39f7fe700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa390009830 tx=0x7fa39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa390009320 con 0x7fa3a00f62a0 2024-02-23T00:28:49.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0106cc0 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa390013380 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a0107150 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa390014420 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.986+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa39001e420 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.987+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a0045ad0 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.987+0000 7fa387fff700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 0x7fa388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.987+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa390041140 con 0x7fa3a00f62a0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.987+0000 7fa39effd700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 0x7fa388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:49.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.987+0000 7fa39effd700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 0x7fa388032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa3940096a0 tx=0x7fa394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:49.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:49.992+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa39000fcd0 con 0x7fa3a00f62a0 2024-02-23T00:28:50.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.200+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3a00fbe30 con 0x7fa3a00f62a0 2024-02-23T00:28:50.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.201+0000 7fa387fff700 1 -- 172.21.15.118:0/1291498548 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa39002c2d0 con 0x7fa3a00f62a0 2024-02-23T00:28:50.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:50.202 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.202+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 msgr2=0x7fa388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.202+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 0x7fa388032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fa3940096a0 tx=0x7fa394006b40).stop 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.202+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 msgr2=0x7fa3a01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.202+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa390009830 tx=0x7fa39000b040).stop 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 shutdown_connections 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa38802ff60 0x7fa388032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 --2- 172.21.15.118:0/1291498548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3a00f62a0 0x7fa3a01065d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 >> 172.21.15.118:0/1291498548 conn(0x7fa3a00f1aa0 msgr2=0x7fa3a00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:50.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 shutdown_connections 2024-02-23T00:28:50.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:50.203+0000 7fa3a4f0e700 1 -- 172.21.15.118:0/1291498548 wait complete. 2024-02-23T00:28:50.205 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:50.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:50.201618+0000 mon.smithi090 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.118:0/1291498548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:51.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:51.161207+0000 mgr.smithi090.pxxmlz (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:51.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:51.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:53.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:53.161487+0000 mgr.smithi090.pxxmlz (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- 172.21.15.118:0/594716725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 msgr2=0x7f12200f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 --2- 172.21.15.118:0/594716725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12200f4be0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1214009fb0 tx=0x7f1214009b30).stop 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- 172.21.15.118:0/594716725 shutdown_connections 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 --2- 172.21.15.118:0/594716725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12200f4be0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- 172.21.15.118:0/594716725 >> 172.21.15.118:0/594716725 conn(0x7f12200f02e0 msgr2=0x7f12200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- 172.21.15.118:0/594716725 shutdown_connections 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- 172.21.15.118:0/594716725 wait complete. 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 Processor -- start 2024-02-23T00:28:54.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- start start 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.960+0000 7f1225a29700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1220108ed0 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f121effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f121effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60940/0 (socket says 172.21.15.118:60940) 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f121effd700 1 -- 172.21.15.118:0/64264449 learned_addr learned my addr 172.21.15.118:0/64264449 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f121effd700 1 -- 172.21.15.118:0/64264449 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214005040 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f121effd700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f12140089f0 tx=0x7f121400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1214009320 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12201090d0 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1214013380 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f121401ed80 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.961+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1220109560 con 0x7f12200f47c0 2024-02-23T00:28:54.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.962+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1214014420 con 0x7f12200f47c0 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.962+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1220045ad0 con 0x7f12200f47c0 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.962+0000 7f12077fe700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 0x7f1208032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.962+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12140410e0 con 0x7f12200f47c0 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.963+0000 7f121e7fc700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 0x7f1208032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.963+0000 7f121e7fc700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 0x7f1208032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f12100096a0 tx=0x7f1210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:54.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:53.967+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f121400faf0 con 0x7f12200f47c0 2024-02-23T00:28:54.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.171+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f122010a3b0 con 0x7f12200f47c0 2024-02-23T00:28:54.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.171+0000 7f12077fe700 1 -- 172.21.15.118:0/64264449 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f121400faf0 con 0x7f12200f47c0 2024-02-23T00:28:54.173 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:54.173 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 msgr2=0x7f1208032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 0x7f1208032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f12100096a0 tx=0x7f1210006b40).stop 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 msgr2=0x7f12201089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f12140089f0 tx=0x7f121400b040).stop 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 shutdown_connections 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f120802ff60 0x7f1208032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 --2- 172.21.15.118:0/64264449 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12200f47c0 0x7f12201089e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 >> 172.21.15.118:0/64264449 conn(0x7f12200f02e0 msgr2=0x7f12200f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 shutdown_connections 2024-02-23T00:28:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:54.173+0000 7f1225a29700 1 -- 172.21.15.118:0/64264449 wait complete. 2024-02-23T00:28:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:54.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:54.172430+0000 mon.smithi090 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.118:0/64264449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:56.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:56.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:28:56.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:54.788833+0000 mon.smithi090 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:28:56.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:28:54.790131+0000 mgr.smithi090.pxxmlz (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi090 on smithi090 2024-02-23T00:28:56.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:55.161749+0000 mgr.smithi090.pxxmlz (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:57.162060+0000 mgr.smithi090.pxxmlz (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:58.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 -- 172.21.15.118:0/1568858978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240f5410 msgr2=0x7fe3240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:58.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1568858978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240f5410 0x7fe3240f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fe318009fb0 tx=0x7fe318009b30).stop 2024-02-23T00:28:58.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 -- 172.21.15.118:0/1568858978 shutdown_connections 2024-02-23T00:28:58.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1568858978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240f5410 0x7fe3240f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 -- 172.21.15.118:0/1568858978 >> 172.21.15.118:0/1568858978 conn(0x7fe3240f0b70 msgr2=0x7fe3240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 -- 172.21.15.118:0/1568858978 shutdown_connections 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.022+0000 7fe32abb1700 1 -- 172.21.15.118:0/1568858978 wait complete. 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32abb1700 1 Processor -- start 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32abb1700 1 -- start start 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32abb1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32abb1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240fd8c0 con 0x7fe3240fd4a0 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32894d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32894d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36212/0 (socket says 172.21.15.118:36212) 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.023+0000 7fe32894d700 1 -- 172.21.15.118:0/1658359748 learned_addr learned my addr 172.21.15.118:0/1658359748 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe32894d700 1 -- 172.21.15.118:0/1658359748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318005040 con 0x7fe3240fd4a0 2024-02-23T00:28:58.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe32894d700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fe3240f6510 tx=0x7fe318009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe318013380 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3240fbe50 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe318013790 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe31801ee70 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.024+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3240fc340 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.025+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe318025060 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.025+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324045ad0 con 0x7fe3240fd4a0 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.025+0000 7fe3217fa700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 0x7fe314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:58.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.025+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe318040e50 con 0x7fe3240fd4a0 2024-02-23T00:28:58.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.025+0000 7fe323fff700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 0x7fe314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:58.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.026+0000 7fe323fff700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 0x7fe314032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe3100096a0 tx=0x7fe310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:58.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.029+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe31801c950 con 0x7fe3240fd4a0 2024-02-23T00:28:58.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.235+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe32410cae0 con 0x7fe3240fd4a0 2024-02-23T00:28:58.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.236+0000 7fe3217fa700 1 -- 172.21.15.118:0/1658359748 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe31802d2d0 con 0x7fe3240fd4a0 2024-02-23T00:28:58.238 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:28:58.238 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:28:58.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 msgr2=0x7fe314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 0x7fe314032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe3100096a0 tx=0x7fe310006b40).stop 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 msgr2=0x7fe3240fb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fe3240f6510 tx=0x7fe318009b30).stop 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 shutdown_connections 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe31402ff60 0x7fe314032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 --2- 172.21.15.118:0/1658359748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3240fd4a0 0x7fe3240fb960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 >> 172.21.15.118:0/1658359748 conn(0x7fe3240f0b70 msgr2=0x7fe3240f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 shutdown_connections 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:28:58.238+0000 7fe32abb1700 1 -- 172.21.15.118:0/1658359748 wait complete. 2024-02-23T00:28:58.240 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:28:58.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:28:58.237145+0000 mon.smithi090 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.118:0/1658359748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:28:59.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:28:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:28:59.162387+0000 mgr.smithi090.pxxmlz (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:28:59.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:28:59.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:01.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:01.162741+0000 mgr.smithi090.pxxmlz (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:02.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 -- 172.21.15.118:0/996709730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 msgr2=0x7f673c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 --2- 172.21.15.118:0/996709730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c0f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f672c009fb0 tx=0x7f672c009b30).stop 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 -- 172.21.15.118:0/996709730 shutdown_connections 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 --2- 172.21.15.118:0/996709730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c0f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 -- 172.21.15.118:0/996709730 >> 172.21.15.118:0/996709730 conn(0x7f673c0f1c70 msgr2=0x7f673c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 -- 172.21.15.118:0/996709730 shutdown_connections 2024-02-23T00:29:02.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.082+0000 7f6741845700 1 -- 172.21.15.118:0/996709730 wait complete. 2024-02-23T00:29:02.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f6741845700 1 Processor -- start 2024-02-23T00:29:02.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f6741845700 1 -- start start 2024-02-23T00:29:02.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f6741845700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f6741845700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c106cd0 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f673affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f673affd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36228/0 (socket says 172.21.15.118:36228) 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f673affd700 1 -- 172.21.15.118:0/2968546792 learned_addr learned my addr 172.21.15.118:0/2968546792 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f673affd700 1 -- 172.21.15.118:0/2968546792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c005040 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.083+0000 7f673affd700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f672c005010 tx=0x7f672c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.084+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c009320 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.084+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f672c013380 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.084+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c106ed0 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.084+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672c014e50 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.084+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c107360 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f672c014420 con 0x7f673c0f6510 2024-02-23T00:29:02.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c045ad0 con 0x7f673c0f6510 2024-02-23T00:29:02.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f67237fe700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 0x7f67240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:02.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f672c040f40 con 0x7f673c0f6510 2024-02-23T00:29:02.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f673a7fc700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 0x7f67240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:02.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.085+0000 7f673a7fc700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 0x7f67240323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f67300096a0 tx=0x7f6730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:02.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.089+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f672c0095f0 con 0x7f673c0f6510 2024-02-23T00:29:02.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.297+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f673c108210 con 0x7f673c0f6510 2024-02-23T00:29:02.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.298+0000 7f67237fe700 1 -- 172.21.15.118:0/2968546792 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f672c01db70 con 0x7f673c0f6510 2024-02-23T00:29:02.299 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:02.300 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:02.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 msgr2=0x7f67240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:02.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 0x7f67240323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f67300096a0 tx=0x7f6730006b40).stop 2024-02-23T00:29:02.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 msgr2=0x7f673c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f672c005010 tx=0x7f672c00b040).stop 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 shutdown_connections 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f672402ff10 0x7f67240323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 --2- 172.21.15.118:0/2968546792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f673c0f6510 0x7f673c1067e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 >> 172.21.15.118:0/2968546792 conn(0x7f673c0f1c70 msgr2=0x7f673c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 shutdown_connections 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:02.300+0000 7f6741845700 1 -- 172.21.15.118:0/2968546792 wait complete. 2024-02-23T00:29:02.302 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:02.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:02.299056+0000 mon.smithi090 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.118:0/2968546792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:03.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:03.163128+0000 mgr.smithi090.pxxmlz (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:04.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:04.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:05.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:05.163519+0000 mgr.smithi090.pxxmlz (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 -- 172.21.15.118:0/4032999975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 msgr2=0x7fd7f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 --2- 172.21.15.118:0/4032999975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f00f6930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fd7e4009fb0 tx=0x7fd7e4009b30).stop 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 -- 172.21.15.118:0/4032999975 shutdown_connections 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 --2- 172.21.15.118:0/4032999975 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f00f6930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 -- 172.21.15.118:0/4032999975 >> 172.21.15.118:0/4032999975 conn(0x7fd7f00f1c70 msgr2=0x7fd7f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:06.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 -- 172.21.15.118:0/4032999975 shutdown_connections 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.034+0000 7fd7f5862700 1 -- 172.21.15.118:0/4032999975 wait complete. 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7f5862700 1 Processor -- start 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7f5862700 1 -- start start 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7f5862700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7f5862700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f0106cd0 con 0x7fd7f00f6510 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53044/0 (socket says 172.21.15.118:53044) 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.035+0000 7fd7eeffd700 1 -- 172.21.15.118:0/2171589822 learned_addr learned my addr 172.21.15.118:0/2171589822 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7eeffd700 1 -- 172.21.15.118:0/2171589822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4005040 con 0x7fd7f00f6510 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7eeffd700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd7e4005010 tx=0x7fd7e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e4009320 con 0x7fd7f00f6510 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f0106ed0 con 0x7fd7f00f6510 2024-02-23T00:29:06.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7e4013380 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e4014e50 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.036+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f0107360 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.037+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f0045ad0 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.037+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7e40095f0 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.037+0000 7fd7d77fe700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 0x7fd7d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.037+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7e4040e90 con 0x7fd7f00f6510 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.037+0000 7fd7ee7fc700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 0x7fd7d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.038+0000 7fd7ee7fc700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 0x7fd7d80323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd7e00096a0 tx=0x7fd7e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:06.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.042+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7e4014420 con 0x7fd7f00f6510 2024-02-23T00:29:06.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.246+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7f0108210 con 0x7fd7f00f6510 2024-02-23T00:29:06.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.246+0000 7fd7d77fe700 1 -- 172.21.15.118:0/2171589822 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd7e4010230 con 0x7fd7f00f6510 2024-02-23T00:29:06.248 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:06.248 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:06.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 msgr2=0x7fd7d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 0x7fd7d80323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd7e00096a0 tx=0x7fd7e0006b40).stop 2024-02-23T00:29:06.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 msgr2=0x7fd7f01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd7e4005010 tx=0x7fd7e400b040).stop 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 shutdown_connections 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd7d802ff10 0x7fd7d80323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 --2- 172.21.15.118:0/2171589822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7f00f6510 0x7fd7f01067e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 >> 172.21.15.118:0/2171589822 conn(0x7fd7f00f1c70 msgr2=0x7fd7f00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 shutdown_connections 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:06.248+0000 7fd7f5862700 1 -- 172.21.15.118:0/2171589822 wait complete. 2024-02-23T00:29:06.250 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:06.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:06.247176+0000 mon.smithi090 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.118:0/2171589822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:07.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:07.163894+0000 mgr.smithi090.pxxmlz (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:07.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:07.969 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:09.164203+0000 mgr.smithi090.pxxmlz (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- 172.21.15.118:0/4176708716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 msgr2=0x7f45540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 --2- 172.21.15.118:0/4176708716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45540f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f4544009fb0 tx=0x7f4544009b30).stop 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- 172.21.15.118:0/4176708716 shutdown_connections 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 --2- 172.21.15.118:0/4176708716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45540f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- 172.21.15.118:0/4176708716 >> 172.21.15.118:0/4176708716 conn(0x7f45540f1c70 msgr2=0x7f45540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- 172.21.15.118:0/4176708716 shutdown_connections 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- 172.21.15.118:0/4176708716 wait complete. 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 Processor -- start 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- start start 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.078+0000 7f455bec6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554106cd0 con 0x7f45540f6510 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f4559c62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f4559c62700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53076/0 (socket says 172.21.15.118:53076) 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f4559c62700 1 -- 172.21.15.118:0/3819828552 learned_addr learned my addr 172.21.15.118:0/3819828552 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f4559c62700 1 -- 172.21.15.118:0/3819828552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544005040 con 0x7f45540f6510 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f4559c62700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4544005010 tx=0x7f454400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544009320 con 0x7f45540f6510 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4554106ed0 con 0x7f45540f6510 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4544013380 con 0x7f45540f6510 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544014e50 con 0x7f45540f6510 2024-02-23T00:29:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.079+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4554107360 con 0x7f45540f6510 2024-02-23T00:29:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.080+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4544014420 con 0x7f45540f6510 2024-02-23T00:29:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.080+0000 7f454a7fc700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 0x7f45400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.080+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4544040e30 con 0x7f45540f6510 2024-02-23T00:29:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.080+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554045ad0 con 0x7f45540f6510 2024-02-23T00:29:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.081+0000 7f4559461700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 0x7f45400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.081+0000 7f4559461700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 0x7f45400323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f45500096a0 tx=0x7f4550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.085+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45440095f0 con 0x7f45540f6510 2024-02-23T00:29:10.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.289+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45540fc190 con 0x7f45540f6510 2024-02-23T00:29:10.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.290+0000 7f454a7fc700 1 -- 172.21.15.118:0/3819828552 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f454402b020 con 0x7f45540f6510 2024-02-23T00:29:10.292 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:10.292 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 msgr2=0x7f45400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 0x7f45400323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f45500096a0 tx=0x7f4550006b40).stop 2024-02-23T00:29:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 msgr2=0x7f45541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f4544005010 tx=0x7f454400b040).stop 2024-02-23T00:29:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 shutdown_connections 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f454002ff10 0x7f45400323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 --2- 172.21.15.118:0/3819828552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45540f6510 0x7f45541067e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 >> 172.21.15.118:0/3819828552 conn(0x7f45540f1c70 msgr2=0x7f45540f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 shutdown_connections 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:10.292+0000 7f455bec6700 1 -- 172.21.15.118:0/3819828552 wait complete. 2024-02-23T00:29:10.294 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:10.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:10.291157+0000 mon.smithi090 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.118:0/3819828552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:11.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:11.164474+0000 mgr.smithi090.pxxmlz (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:11.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:11.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:13.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:13.164793+0000 mgr.smithi090.pxxmlz (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:13.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 -- 172.21.15.118:0/2232460519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 msgr2=0x7fbf300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 --2- 172.21.15.118:0/2232460519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf300f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fbf24009fb0 tx=0x7fbf24009b30).stop 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 -- 172.21.15.118:0/2232460519 shutdown_connections 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 --2- 172.21.15.118:0/2232460519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf300f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 -- 172.21.15.118:0/2232460519 >> 172.21.15.118:0/2232460519 conn(0x7fbf300f0b70 msgr2=0x7fbf300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 -- 172.21.15.118:0/2232460519 shutdown_connections 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 -- 172.21.15.118:0/2232460519 wait complete. 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.888+0000 7fbf35d32700 1 Processor -- start 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf35d32700 1 -- start start 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf35d32700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf35d32700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30060220 con 0x7fbf300f5410 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53092/0 (socket says 172.21.15.118:53092) 2024-02-23T00:29:13.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf2f7fe700 1 -- 172.21.15.118:0/409380988 learned_addr learned my addr 172.21.15.118:0/409380988 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf2f7fe700 1 -- 172.21.15.118:0/409380988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24005040 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf2f7fe700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbf24005010 tx=0x7fbf2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.889+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24009320 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf30060420 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf24013380 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24014e50 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf30109bb0 con 0x7fbf300f5410 2024-02-23T00:29:13.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf240263a0 con 0x7fbf300f5410 2024-02-23T00:29:13.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf30059100 con 0x7fbf300f5410 2024-02-23T00:29:13.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.890+0000 7fbf17fff700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 0x7fbf180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:13.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.891+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf24040ee0 con 0x7fbf300f5410 2024-02-23T00:29:13.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.891+0000 7fbf2effd700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 0x7fbf180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:13.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.891+0000 7fbf2effd700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 0x7fbf180323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbf200096a0 tx=0x7fbf20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:13.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:13.895+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf2401dd70 con 0x7fbf300f5410 2024-02-23T00:29:14.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.099+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf300fafa0 con 0x7fbf300f5410 2024-02-23T00:29:14.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.100+0000 7fbf17fff700 1 -- 172.21.15.118:0/409380988 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf2401d980 con 0x7fbf300f5410 2024-02-23T00:29:14.101 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:14.102 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 msgr2=0x7fbf180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 0x7fbf180323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbf200096a0 tx=0x7fbf20006b40).stop 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 msgr2=0x7fbf30109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbf24005010 tx=0x7fbf2400b040).stop 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 shutdown_connections 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fbf1802ff10 0x7fbf180323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:14.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 --2- 172.21.15.118:0/409380988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf300f5410 0x7fbf30109500 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:14.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 >> 172.21.15.118:0/409380988 conn(0x7fbf300f0b70 msgr2=0x7fbf300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:14.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 shutdown_connections 2024-02-23T00:29:14.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:14.101+0000 7fbf35d32700 1 -- 172.21.15.118:0/409380988 wait complete. 2024-02-23T00:29:14.104 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:14.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:14.100809+0000 mon.smithi090 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.118:0/409380988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:15.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:15.165118+0000 mgr.smithi090.pxxmlz (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:15.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:15.836 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:17.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:17.165516+0000 mgr.smithi090.pxxmlz (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:18.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 -- 172.21.15.118:0/2300105287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 msgr2=0x7fea200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:18.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 --2- 172.21.15.118:0/2300105287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea200f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fea10009fb0 tx=0x7fea10009b30).stop 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 -- 172.21.15.118:0/2300105287 shutdown_connections 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 --2- 172.21.15.118:0/2300105287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea200f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 -- 172.21.15.118:0/2300105287 >> 172.21.15.118:0/2300105287 conn(0x7fea200f0b70 msgr2=0x7fea200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 -- 172.21.15.118:0/2300105287 shutdown_connections 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.097+0000 7fea2877d700 1 -- 172.21.15.118:0/2300105287 wait complete. 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea2877d700 1 Processor -- start 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea2877d700 1 -- start start 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea2877d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea2877d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20060220 con 0x7fea200f5410 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea26519700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea26519700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35046/0 (socket says 172.21.15.118:35046) 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.098+0000 7fea26519700 1 -- 172.21.15.118:0/4109881940 learned_addr learned my addr 172.21.15.118:0/4109881940 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea26519700 1 -- 172.21.15.118:0/4109881940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10005040 con 0x7fea200f5410 2024-02-23T00:29:18.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea26519700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fea10005010 tx=0x7fea1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:18.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea10009320 con 0x7fea200f5410 2024-02-23T00:29:18.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea20060420 con 0x7fea200f5410 2024-02-23T00:29:18.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea10013380 con 0x7fea200f5410 2024-02-23T00:29:18.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea10014e50 con 0x7fea200f5410 2024-02-23T00:29:18.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.099+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea20109bb0 con 0x7fea200f5410 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.100+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea20059100 con 0x7fea200f5410 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.100+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea100095f0 con 0x7fea200f5410 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.100+0000 7fea16ffd700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 0x7fea0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.100+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea10040e90 con 0x7fea200f5410 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.100+0000 7fea25d18700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 0x7fea0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.101+0000 7fea25d18700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 0x7fea0c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fea1c0096a0 tx=0x7fea1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:18.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.104+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea10014420 con 0x7fea200f5410 2024-02-23T00:29:18.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.307+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea20045ad0 con 0x7fea200f5410 2024-02-23T00:29:18.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.308+0000 7fea16ffd700 1 -- 172.21.15.118:0/4109881940 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea10010230 con 0x7fea200f5410 2024-02-23T00:29:18.310 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:18.310 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 msgr2=0x7fea0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 0x7fea0c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fea1c0096a0 tx=0x7fea1c006b40).stop 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 msgr2=0x7fea20109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fea10005010 tx=0x7fea1000b040).stop 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 shutdown_connections 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fea0c02ff10 0x7fea0c0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 --2- 172.21.15.118:0/4109881940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea200f5410 0x7fea20109500 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 >> 172.21.15.118:0/4109881940 conn(0x7fea200f0b70 msgr2=0x7fea200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 shutdown_connections 2024-02-23T00:29:18.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:18.309+0000 7fea2877d700 1 -- 172.21.15.118:0/4109881940 wait complete. 2024-02-23T00:29:18.313 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:18.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:18.309616+0000 mon.smithi090 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.118:0/4109881940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:19.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:19.165844+0000 mgr.smithi090.pxxmlz (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:20.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:20.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:21.166172+0000 mgr.smithi090.pxxmlz (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:22.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 -- 172.21.15.118:0/2317244452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 msgr2=0x7fecfc0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 --2- 172.21.15.118:0/2317244452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc0fb570 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fecf0009fb0 tx=0x7fecf0009b30).stop 2024-02-23T00:29:22.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 -- 172.21.15.118:0/2317244452 shutdown_connections 2024-02-23T00:29:22.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 --2- 172.21.15.118:0/2317244452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc0fb570 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 -- 172.21.15.118:0/2317244452 >> 172.21.15.118:0/2317244452 conn(0x7fecfc0f0b70 msgr2=0x7fecfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 -- 172.21.15.118:0/2317244452 shutdown_connections 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.081+0000 7fed02b80700 1 -- 172.21.15.118:0/2317244452 wait complete. 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed02b80700 1 Processor -- start 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed02b80700 1 -- start start 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed02b80700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed02b80700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc108e60 con 0x7fecfc060200 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed0091c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed0091c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35056/0 (socket says 172.21.15.118:35056) 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.082+0000 7fed0091c700 1 -- 172.21.15.118:0/910604739 learned_addr learned my addr 172.21.15.118:0/910604739 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fed0091c700 1 -- 172.21.15.118:0/910604739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0005040 con 0x7fecfc060200 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fed0091c700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fecf00089f0 tx=0x7fecf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf0009320 con 0x7fecfc060200 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecf0013380 con 0x7fecfc060200 2024-02-23T00:29:22.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc109060 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf0014e20 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.083+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc1094f0 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.084+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecf00263a0 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.084+0000 7fecf97fa700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 0x7fecec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.084+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecf0040eb0 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.084+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc045ad0 con 0x7fecfc060200 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.084+0000 7fecfbfff700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 0x7fecec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.085+0000 7fecfbfff700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 0x7fecec0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fece80096a0 tx=0x7fece8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:22.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.089+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecf0012920 con 0x7fecfc060200 2024-02-23T00:29:22.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.297+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecfc0fe380 con 0x7fecfc060200 2024-02-23T00:29:22.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.297+0000 7fecf97fa700 1 -- 172.21.15.118:0/910604739 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecf002b020 con 0x7fecfc060200 2024-02-23T00:29:22.300 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:22.300 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 msgr2=0x7fecec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 0x7fecec0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fece80096a0 tx=0x7fece8006b40).stop 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 msgr2=0x7fecfc108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fecf00089f0 tx=0x7fecf000b040).stop 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 shutdown_connections 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fecec02ff10 0x7fecec0323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 --2- 172.21.15.118:0/910604739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecfc060200 0x7fecfc108970 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 >> 172.21.15.118:0/910604739 conn(0x7fecfc0f0b70 msgr2=0x7fecfc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:22.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 shutdown_connections 2024-02-23T00:29:22.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:22.299+0000 7fed02b80700 1 -- 172.21.15.118:0/910604739 wait complete. 2024-02-23T00:29:22.303 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:22.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:22.299312+0000 mon.smithi090 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.118:0/910604739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:23.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:23.166563+0000 mgr.smithi090.pxxmlz (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:24.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:24.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:25.166987+0000 mgr.smithi090.pxxmlz (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:25.248218+0000 mon.smithi090 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.951+0000 7f992c215700 1 -- 172.21.15.118:0/1948463131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 msgr2=0x7f99240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.951+0000 7f992c215700 1 --2- 172.21.15.118:0/1948463131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99240f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f9918009fb0 tx=0x7f9918009b30).stop 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- 172.21.15.118:0/1948463131 shutdown_connections 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 --2- 172.21.15.118:0/1948463131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99240f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- 172.21.15.118:0/1948463131 >> 172.21.15.118:0/1948463131 conn(0x7f99240f1c70 msgr2=0x7f99240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- 172.21.15.118:0/1948463131 shutdown_connections 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- 172.21.15.118:0/1948463131 wait complete. 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 Processor -- start 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- start start 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f992c215700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9924108ed0 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.952+0000 7f9929fb1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f9929fb1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36102/0 (socket says 172.21.15.118:36102) 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f9929fb1700 1 -- 172.21.15.118:0/2900748557 learned_addr learned my addr 172.21.15.118:0/2900748557 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f9929fb1700 1 -- 172.21.15.118:0/2900748557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918005040 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f9929fb1700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9918005010 tx=0x7f991800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918009320 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99241090d0 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9918013380 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9918014e50 con 0x7f99240f6510 2024-02-23T00:29:25.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.953+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9924109560 con 0x7f99240f6510 2024-02-23T00:29:25.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.954+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99180263a0 con 0x7f99240f6510 2024-02-23T00:29:25.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.954+0000 7f99167fc700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.954+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9918040f70 con 0x7f99240f6510 2024-02-23T00:29:25.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.954+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9924045ad0 con 0x7f99240f6510 2024-02-23T00:29:25.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.954+0000 7f99297b0700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.955+0000 7f99297b0700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 0x7f99100323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:25.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:25.960+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9918012920 con 0x7f99240f6510 2024-02-23T00:29:26.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.162+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f992410a280 con 0x7f99240f6510 2024-02-23T00:29:26.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.163+0000 7f99167fc700 1 -- 172.21.15.118:0/2900748557 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f991802b020 con 0x7f99240f6510 2024-02-23T00:29:26.165 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:26.165 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 msgr2=0x7f99100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 0x7f99100323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).stop 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 msgr2=0x7f99241089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9918005010 tx=0x7f991800b040).stop 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 shutdown_connections 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f991002ff10 0x7f99100323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 --2- 172.21.15.118:0/2900748557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99240f6510 0x7f99241089e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 >> 172.21.15.118:0/2900748557 conn(0x7f99240f1c70 msgr2=0x7f99240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 shutdown_connections 2024-02-23T00:29:26.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:26.164+0000 7f992c215700 1 -- 172.21.15.118:0/2900748557 wait complete. 2024-02-23T00:29:26.168 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:26.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:25.255334+0000 mon.smithi090 (mon.0) 217 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:29:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:26.164616+0000 mon.smithi090 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.118:0/2900748557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:27.167336+0000 mgr.smithi090.pxxmlz (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:27.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:27.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:29.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:29.167672+0000 mgr.smithi090.pxxmlz (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- 172.21.15.118:0/2197432575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 msgr2=0x7f99f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 --2- 172.21.15.118:0/2197432575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40f55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f99f0009fb0 tx=0x7f99f0009b30).stop 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- 172.21.15.118:0/2197432575 shutdown_connections 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 --2- 172.21.15.118:0/2197432575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40f55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- 172.21.15.118:0/2197432575 >> 172.21.15.118:0/2197432575 conn(0x7f99f40f09a0 msgr2=0x7f99f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- 172.21.15.118:0/2197432575 shutdown_connections 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- 172.21.15.118:0/2197432575 wait complete. 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 Processor -- start 2024-02-23T00:29:30.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- start start 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.098+0000 7f99fa790700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40fb6f0 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99f978e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99f978e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36120/0 (socket says 172.21.15.118:36120) 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99f978e700 1 -- 172.21.15.118:0/3097564260 learned_addr learned my addr 172.21.15.118:0/3097564260 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99f978e700 1 -- 172.21.15.118:0/3097564260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0005040 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99f978e700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f99f00089f0 tx=0x7f99f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f0009320 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f40fb890 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99f0013380 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99f0014e20 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.099+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f40fbd20 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.100+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99f00095f0 con 0x7f99f40f51a0 2024-02-23T00:29:30.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.100+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f4059100 con 0x7f99f40f51a0 2024-02-23T00:29:30.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.100+0000 7f99e9ffb700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 0x7f99e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:30.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.100+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99f0040f50 con 0x7f99f40f51a0 2024-02-23T00:29:30.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.101+0000 7f99f8f8d700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 0x7f99e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:30.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.101+0000 7f99f8f8d700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 0x7f99e00323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f99e40096a0 tx=0x7f99e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:30.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.105+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99f0014420 con 0x7f99f40f51a0 2024-02-23T00:29:30.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.309+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99f40fad30 con 0x7f99f40f51a0 2024-02-23T00:29:30.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.309+0000 7f99e9ffb700 1 -- 172.21.15.118:0/3097564260 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99f001d980 con 0x7f99f40f51a0 2024-02-23T00:29:30.311 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:30.311 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:30.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 msgr2=0x7f99e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:30.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 0x7f99e00323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f99e40096a0 tx=0x7f99e4006b40).stop 2024-02-23T00:29:30.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 msgr2=0x7f99f40fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f99f00089f0 tx=0x7f99f000b040).stop 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 shutdown_connections 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f99e002ff10 0x7f99e00323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 --2- 172.21.15.118:0/3097564260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99f40f51a0 0x7f99f40fd270 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 >> 172.21.15.118:0/3097564260 conn(0x7f99f40f09a0 msgr2=0x7f99f40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 shutdown_connections 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:30.311+0000 7f99fa790700 1 -- 172.21.15.118:0/3097564260 wait complete. 2024-02-23T00:29:30.313 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:30.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:30.310314+0000 mon.smithi090 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.118:0/3097564260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:31.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:31.167977+0000 mgr.smithi090.pxxmlz (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:32.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:32.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:33.168266+0000 mgr.smithi090.pxxmlz (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 -- 172.21.15.118:0/150627321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 msgr2=0x7f64d80f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 --2- 172.21.15.118:0/150627321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d80f6ee0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f64c8009fb0 tx=0x7f64c8009b30).stop 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 -- 172.21.15.118:0/150627321 shutdown_connections 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 --2- 172.21.15.118:0/150627321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d80f6ee0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 -- 172.21.15.118:0/150627321 >> 172.21.15.118:0/150627321 conn(0x7f64d80f0b70 msgr2=0x7f64d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 -- 172.21.15.118:0/150627321 shutdown_connections 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.092+0000 7f64df782700 1 -- 172.21.15.118:0/150627321 wait complete. 2024-02-23T00:29:34.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64df782700 1 Processor -- start 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64df782700 1 -- start start 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64df782700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64df782700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d810d2c0 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64dd51e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64dd51e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:36136/0 (socket says 172.21.15.118:36136) 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.093+0000 7f64dd51e700 1 -- 172.21.15.118:0/1272836120 learned_addr learned my addr 172.21.15.118:0/1272836120 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64dd51e700 1 -- 172.21.15.118:0/1272836120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8005040 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64dd51e700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f64c80089f0 tx=0x7f64c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c8009320 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d810d4c0 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64c8013380 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c8014e20 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.094+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d810e100 con 0x7f64d8060280 2024-02-23T00:29:34.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.095+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64c80263a0 con 0x7f64d8060280 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.095+0000 7f64cdffb700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 0x7f64c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.095+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64c8041190 con 0x7f64d8060280 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.095+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d810d950 con 0x7f64d8060280 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.095+0000 7f64dcd1d700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 0x7f64c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.096+0000 7f64dcd1d700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 0x7f64c40323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f64d40096a0 tx=0x7f64d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:34.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.100+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64c801ccf0 con 0x7f64d8060280 2024-02-23T00:29:34.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.309+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64d810e760 con 0x7f64d8060280 2024-02-23T00:29:34.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.309+0000 7f64cdffb700 1 -- 172.21.15.118:0/1272836120 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64c8012330 con 0x7f64d8060280 2024-02-23T00:29:34.311 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:34.311 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:34.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 msgr2=0x7f64c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:34.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 0x7f64c40323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f64d40096a0 tx=0x7f64d4006b40).stop 2024-02-23T00:29:34.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 msgr2=0x7f64d810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:34.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f64c80089f0 tx=0x7f64c800b040).stop 2024-02-23T00:29:34.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 shutdown_connections 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f64c402ff10 0x7f64c40323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 --2- 172.21.15.118:0/1272836120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64d8060280 0x7f64d810cdd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 >> 172.21.15.118:0/1272836120 conn(0x7f64d80f0b70 msgr2=0x7f64d80f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 shutdown_connections 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:34.311+0000 7f64df782700 1 -- 172.21.15.118:0/1272836120 wait complete. 2024-02-23T00:29:34.317 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:34.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:34.310236+0000 mon.smithi090 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.118:0/1272836120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:35.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:35.168599+0000 mgr.smithi090.pxxmlz (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:36.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:36.099 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:37.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:37.169068+0000 mgr.smithi090.pxxmlz (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:38.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 -- 172.21.15.118:0/2875035961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 msgr2=0x7fd9280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 --2- 172.21.15.118:0/2875035961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9280f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fd910009fb0 tx=0x7fd910009b30).stop 2024-02-23T00:29:38.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 -- 172.21.15.118:0/2875035961 shutdown_connections 2024-02-23T00:29:38.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 --2- 172.21.15.118:0/2875035961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9280f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 -- 172.21.15.118:0/2875035961 >> 172.21.15.118:0/2875035961 conn(0x7fd9280f0b70 msgr2=0x7fd9280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 -- 172.21.15.118:0/2875035961 shutdown_connections 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.234+0000 7fd92caea700 1 -- 172.21.15.118:0/2875035961 wait complete. 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92caea700 1 Processor -- start 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92caea700 1 -- start start 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92caea700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92caea700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd928106cc0 con 0x7fd9280f5410 2024-02-23T00:29:38.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39174/0 (socket says 172.21.15.118:39174) 2024-02-23T00:29:38.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.235+0000 7fd92659c700 1 -- 172.21.15.118:0/2935386858 learned_addr learned my addr 172.21.15.118:0/2935386858 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:38.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd92659c700 1 -- 172.21.15.118:0/2935386858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910005040 con 0x7fd9280f5410 2024-02-23T00:29:38.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd92659c700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd910005010 tx=0x7fd91000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:38.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd910009320 con 0x7fd9280f5410 2024-02-23T00:29:38.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd928106ec0 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd910013380 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd910014e50 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.236+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd928107350 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9100095f0 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd928045ad0 con 0x7fd9280f5410 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd91effd700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 0x7fd9140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd910040e90 con 0x7fd9280f5410 2024-02-23T00:29:38.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd925d9b700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 0x7fd9140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.237+0000 7fd925d9b700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 0x7fd9140323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd9180096a0 tx=0x7fd918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:38.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.241+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd910014420 con 0x7fd9280f5410 2024-02-23T00:29:38.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.449+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd928108200 con 0x7fd9280f5410 2024-02-23T00:29:38.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.450+0000 7fd91effd700 1 -- 172.21.15.118:0/2935386858 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd910010230 con 0x7fd9280f5410 2024-02-23T00:29:38.451 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:38.451 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:38.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 msgr2=0x7fd9140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 0x7fd9140323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd9180096a0 tx=0x7fd918006b40).stop 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 msgr2=0x7fd9281067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd910005010 tx=0x7fd91000b040).stop 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 shutdown_connections 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd91402ff10 0x7fd9140323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 --2- 172.21.15.118:0/2935386858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9280f5410 0x7fd9281067d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 >> 172.21.15.118:0/2935386858 conn(0x7fd9280f0b70 msgr2=0x7fd92805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 shutdown_connections 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:38.451+0000 7fd92caea700 1 -- 172.21.15.118:0/2935386858 wait complete. 2024-02-23T00:29:38.453 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:38.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:38.450534+0000 mon.smithi090 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.118:0/2935386858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:39.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:39.169423+0000 mgr.smithi090.pxxmlz (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:40.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:40.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:41.169785+0000 mgr.smithi090.pxxmlz (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:42.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 -- 172.21.15.118:0/2073082900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 msgr2=0x7fb2540f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:42.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2073082900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2540f68a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fb244009fb0 tx=0x7fb244009b30).stop 2024-02-23T00:29:42.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 -- 172.21.15.118:0/2073082900 shutdown_connections 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2073082900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2540f68a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 -- 172.21.15.118:0/2073082900 >> 172.21.15.118:0/2073082900 conn(0x7fb2540f1c00 msgr2=0x7fb2540f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 -- 172.21.15.118:0/2073082900 shutdown_connections 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.344+0000 7fb25ba60700 1 -- 172.21.15.118:0/2073082900 wait complete. 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb25ba60700 1 Processor -- start 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb25ba60700 1 -- start start 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb25ba60700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb25ba60700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb254106ce0 con 0x7fb2540f6480 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb2597fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb2597fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39190/0 (socket says 172.21.15.118:39190) 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.345+0000 7fb2597fc700 1 -- 172.21.15.118:0/2828718491 learned_addr learned my addr 172.21.15.118:0/2828718491 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb2597fc700 1 -- 172.21.15.118:0/2828718491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244005040 con 0x7fb2540f6480 2024-02-23T00:29:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb2597fc700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb244005010 tx=0x7fb24400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:42.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb244009320 con 0x7fb2540f6480 2024-02-23T00:29:42.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb254106ee0 con 0x7fb2540f6480 2024-02-23T00:29:42.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb244013380 con 0x7fb2540f6480 2024-02-23T00:29:42.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb244014e50 con 0x7fb2540f6480 2024-02-23T00:29:42.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.346+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb254107370 con 0x7fb2540f6480 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.347+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2440095f0 con 0x7fb2540f6480 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.347+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb254045ad0 con 0x7fb2540f6480 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.347+0000 7fb249ffb700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 0x7fb2400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.347+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb244040e90 con 0x7fb2540f6480 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.347+0000 7fb258ffb700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 0x7fb2400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:42.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.348+0000 7fb258ffb700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 0x7fb2400323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb2500096a0 tx=0x7fb250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:42.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.352+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb244014420 con 0x7fb2540f6480 2024-02-23T00:29:42.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.558+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2540fc090 con 0x7fb2540f6480 2024-02-23T00:29:42.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.559+0000 7fb249ffb700 1 -- 172.21.15.118:0/2828718491 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb244010230 con 0x7fb2540f6480 2024-02-23T00:29:42.560 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:42.561 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:42.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.560+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 msgr2=0x7fb2400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:42.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.560+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 0x7fb2400323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb2500096a0 tx=0x7fb250006b40).stop 2024-02-23T00:29:42.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.560+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 msgr2=0x7fb2541067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:42.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.560+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fb244005010 tx=0x7fb24400b040).stop 2024-02-23T00:29:42.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 shutdown_connections 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb24002ff10 0x7fb2400323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 --2- 172.21.15.118:0/2828718491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2540f6480 0x7fb2541067f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 >> 172.21.15.118:0/2828718491 conn(0x7fb2540f1c00 msgr2=0x7fb2540f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 shutdown_connections 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:42.561+0000 7fb25ba60700 1 -- 172.21.15.118:0/2828718491 wait complete. 2024-02-23T00:29:42.563 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:43.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:42.559852+0000 mon.smithi090 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.118:0/2828718491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:44.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:43.170148+0000 mgr.smithi090.pxxmlz (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:44.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:44.294 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:45.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:45.170461+0000 mgr.smithi090.pxxmlz (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 -- 172.21.15.118:0/2133651197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 msgr2=0x7f2abc0f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/2133651197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc0f55e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f2ab0009fb0 tx=0x7f2ab0009b30).stop 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 -- 172.21.15.118:0/2133651197 shutdown_connections 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/2133651197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc0f55e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 -- 172.21.15.118:0/2133651197 >> 172.21.15.118:0/2133651197 conn(0x7f2abc0f0920 msgr2=0x7f2abc0f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:46.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 -- 172.21.15.118:0/2133651197 shutdown_connections 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.353+0000 7f2ac13a3700 1 -- 172.21.15.118:0/2133651197 wait complete. 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2ac13a3700 1 Processor -- start 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2ac13a3700 1 -- start start 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2ac13a3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2ac13a3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc105a30 con 0x7f2abc0f51c0 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2abaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2abaffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47738/0 (socket says 172.21.15.118:47738) 2024-02-23T00:29:46.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.354+0000 7f2abaffd700 1 -- 172.21.15.118:0/1920252704 learned_addr learned my addr 172.21.15.118:0/1920252704 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2abaffd700 1 -- 172.21.15.118:0/1920252704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0005040 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2abaffd700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f2ab0005010 tx=0x7f2ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0009320 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc105c30 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ab0013380 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0014e50 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.355+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc1060c0 con 0x7f2abc0f51c0 2024-02-23T00:29:46.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.356+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ab00095f0 con 0x7f2abc0f51c0 2024-02-23T00:29:46.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.356+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc045ad0 con 0x7f2abc0f51c0 2024-02-23T00:29:46.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.356+0000 7f2aa37fe700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 0x7f2aa40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:46.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.356+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ab0040e70 con 0x7f2abc0f51c0 2024-02-23T00:29:46.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.356+0000 7f2aba7fc700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 0x7f2aa40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:46.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.357+0000 7f2aba7fc700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 0x7f2aa40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2aac0096a0 tx=0x7f2aac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:46.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.360+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ab0014420 con 0x7f2abc0f51c0 2024-02-23T00:29:46.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.567+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2abc106ed0 con 0x7f2abc0f51c0 2024-02-23T00:29:46.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.568+0000 7f2aa37fe700 1 -- 172.21.15.118:0/1920252704 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ab0010230 con 0x7f2abc0f51c0 2024-02-23T00:29:46.570 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:46.570 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:46.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 msgr2=0x7f2aa40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 0x7f2aa40323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2aac0096a0 tx=0x7f2aac006b40).stop 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 msgr2=0x7f2abc105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f2ab0005010 tx=0x7f2ab000b040).stop 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 shutdown_connections 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2aa402ff10 0x7f2aa40323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 --2- 172.21.15.118:0/1920252704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2abc0f51c0 0x7f2abc105540 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:46.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 >> 172.21.15.118:0/1920252704 conn(0x7f2abc0f0920 msgr2=0x7f2abc0f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:46.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 shutdown_connections 2024-02-23T00:29:46.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:46.570+0000 7f2ac13a3700 1 -- 172.21.15.118:0/1920252704 wait complete. 2024-02-23T00:29:46.573 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:47.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:46.568850+0000 mon.smithi090 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.118:0/1920252704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:48.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:47.170813+0000 mgr.smithi090.pxxmlz (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:48.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:48.192 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:49.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:49.171202+0000 mgr.smithi090.pxxmlz (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:50.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 -- 172.21.15.118:0/1127785508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 msgr2=0x7fa1380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:50.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1127785508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1380f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fa128009fb0 tx=0x7fa128009b30).stop 2024-02-23T00:29:50.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 -- 172.21.15.118:0/1127785508 shutdown_connections 2024-02-23T00:29:50.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1127785508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1380f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:50.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 -- 172.21.15.118:0/1127785508 >> 172.21.15.118:0/1127785508 conn(0x7fa1380f0b70 msgr2=0x7fa1380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 -- 172.21.15.118:0/1127785508 shutdown_connections 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.134+0000 7fa13e66d700 1 -- 172.21.15.118:0/1127785508 wait complete. 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa13e66d700 1 Processor -- start 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa13e66d700 1 -- start start 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa13e66d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa13e66d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa138106cc0 con 0x7fa1380f5410 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa137fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa137fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47752/0 (socket says 172.21.15.118:47752) 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.135+0000 7fa137fff700 1 -- 172.21.15.118:0/1204211585 learned_addr learned my addr 172.21.15.118:0/1204211585 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa137fff700 1 -- 172.21.15.118:0/1204211585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128005040 con 0x7fa1380f5410 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa137fff700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa1280089f0 tx=0x7fa12800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:50.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa128009320 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa138106ec0 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa128013380 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa12801ed80 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.136+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa138107350 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa128014420 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa134ff9700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 0x7fa120032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa138045ad0 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa128040fc0 con 0x7fa1380f5410 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa1377fe700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 0x7fa120032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.137+0000 7fa1377fe700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 0x7fa120032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa12c0096a0 tx=0x7fa12c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:50.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.141+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa12801cb10 con 0x7fa1380f5410 2024-02-23T00:29:50.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.346+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa138108200 con 0x7fa1380f5410 2024-02-23T00:29:50.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.347+0000 7fa134ff9700 1 -- 172.21.15.118:0/1204211585 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa128004490 con 0x7fa1380f5410 2024-02-23T00:29:50.349 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:50.349 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 msgr2=0x7fa120032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 0x7fa120032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa12c0096a0 tx=0x7fa12c006b40).stop 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 msgr2=0x7fa1381067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa1280089f0 tx=0x7fa12800b040).stop 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 shutdown_connections 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa12002ff60 0x7fa120032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 --2- 172.21.15.118:0/1204211585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380f5410 0x7fa1381067d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 >> 172.21.15.118:0/1204211585 conn(0x7fa1380f0b70 msgr2=0x7fa13805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 shutdown_connections 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:50.349+0000 7fa13e66d700 1 -- 172.21.15.118:0/1204211585 wait complete. 2024-02-23T00:29:50.351 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:50.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:50.348246+0000 mon.smithi090 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.118:0/1204211585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:51.171652+0000 mgr.smithi090.pxxmlz (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:52.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:52.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:53.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:53.171970+0000 mgr.smithi090.pxxmlz (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- 172.21.15.118:0/1320697157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 msgr2=0x7f91800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 --2- 172.21.15.118:0/1320697157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f917c009fb0 tx=0x7f917c009b30).stop 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- 172.21.15.118:0/1320697157 shutdown_connections 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 --2- 172.21.15.118:0/1320697157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- 172.21.15.118:0/1320697157 >> 172.21.15.118:0/1320697157 conn(0x7f91800f09a0 msgr2=0x7f91800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- 172.21.15.118:0/1320697157 shutdown_connections 2024-02-23T00:29:54.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- 172.21.15.118:0/1320697157 wait complete. 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 Processor -- start 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.233+0000 7f9186c3a700 1 -- start start 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9186c3a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9186c3a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91800f13e0 con 0x7f91800f51a0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9185c38700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9185c38700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47762/0 (socket says 172.21.15.118:47762) 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9185c38700 1 -- 172.21.15.118:0/4009398874 learned_addr learned my addr 172.21.15.118:0/4009398874 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9185c38700 1 -- 172.21.15.118:0/4009398874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c005040 con 0x7f91800f51a0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9185c38700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f917c0089f0 tx=0x7f917c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f917c009320 con 0x7f91800f51a0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91800f1580 con 0x7f91800f51a0 2024-02-23T00:29:54.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f917c013380 con 0x7f91800f51a0 2024-02-23T00:29:54.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f917c014e20 con 0x7f91800f51a0 2024-02-23T00:29:54.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.234+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91800f1a10 con 0x7f91800f51a0 2024-02-23T00:29:54.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.235+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f917c0095f0 con 0x7f91800f51a0 2024-02-23T00:29:54.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.235+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180059100 con 0x7f91800f51a0 2024-02-23T00:29:54.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.235+0000 7f91767fc700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 0x7f916c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:54.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.235+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f917c040ef0 con 0x7f91800f51a0 2024-02-23T00:29:54.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.236+0000 7f9185437700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 0x7f916c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:54.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.236+0000 7f9185437700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 0x7f916c0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f91700096a0 tx=0x7f9170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:54.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.240+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f917c014420 con 0x7f91800f51a0 2024-02-23T00:29:54.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.449+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9180045ad0 con 0x7f91800f51a0 2024-02-23T00:29:54.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.449+0000 7f91767fc700 1 -- 172.21.15.118:0/4009398874 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f917c010230 con 0x7f91800f51a0 2024-02-23T00:29:54.451 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:54.451 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 msgr2=0x7f916c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 0x7f916c0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f91700096a0 tx=0x7f9170006b40).stop 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 msgr2=0x7f91800f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f917c0089f0 tx=0x7f917c00b040).stop 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 shutdown_connections 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f916c02ff10 0x7f916c0323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 --2- 172.21.15.118:0/4009398874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91800f51a0 0x7f91800f2f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 >> 172.21.15.118:0/4009398874 conn(0x7f91800f09a0 msgr2=0x7f918005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 shutdown_connections 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:54.451+0000 7f9186c3a700 1 -- 172.21.15.118:0/4009398874 wait complete. 2024-02-23T00:29:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:54.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:54.450267+0000 mon.smithi090 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.118:0/4009398874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:55.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:55.172296+0000 mgr.smithi090.pxxmlz (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:56.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:29:56.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:29:57.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:57.172613+0000 mgr.smithi090.pxxmlz (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:29:58.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.260+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2234639233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 msgr2=0x7f6cb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.260+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2234639233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f6ca0009fb0 tx=0x7f6ca0009b30).stop 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.260+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2234639233 shutdown_connections 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.260+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2234639233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.260+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2234639233 >> 172.21.15.118:0/2234639233 conn(0x7f6cb00f0b70 msgr2=0x7f6cb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2234639233 shutdown_connections 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2234639233 wait complete. 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 Processor -- start 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 -- start start 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.261+0000 7f6cb56aa700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00fb960 con 0x7f6cb00f5410 2024-02-23T00:29:58.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6caeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6caeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:43840/0 (socket says 172.21.15.118:43840) 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6caeffd700 1 -- 172.21.15.118:0/2805788534 learned_addr learned my addr 172.21.15.118:0/2805788534 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6caeffd700 1 -- 172.21.15.118:0/2805788534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0005040 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6caeffd700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f6ca0005010 tx=0x7f6ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca0009320 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb00fbb00 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ca0013380 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca0014e50 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.262+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb00fbf90 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.263+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ca00095f0 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.263+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb0059100 con 0x7f6cb00f5410 2024-02-23T00:29:58.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.263+0000 7f6ca77fe700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 0x7f6c9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:58.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.263+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ca0040e90 con 0x7f6cb00f5410 2024-02-23T00:29:58.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.263+0000 7f6cae7fc700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 0x7f6c9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:58.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.264+0000 7f6cae7fc700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 0x7f6c9c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6c980096a0 tx=0x7f6c98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:58.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.268+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ca0014420 con 0x7f6cb00f5410 2024-02-23T00:29:58.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.471+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cb00fafa0 con 0x7f6cb00f5410 2024-02-23T00:29:58.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.471+0000 7f6ca77fe700 1 -- 172.21.15.118:0/2805788534 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ca0010230 con 0x7f6cb00f5410 2024-02-23T00:29:58.473 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:29:58.473 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:29:58.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 msgr2=0x7f6c9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:58.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 0x7f6c9c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6c980096a0 tx=0x7f6c98006b40).stop 2024-02-23T00:29:58.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 msgr2=0x7f6cb00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:58.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f6ca0005010 tx=0x7f6ca000b040).stop 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 shutdown_connections 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6c9c02ff10 0x7f6c9c0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 --2- 172.21.15.118:0/2805788534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb00f5410 0x7f6cb00fd4e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 >> 172.21.15.118:0/2805788534 conn(0x7f6cb00f0b70 msgr2=0x7f6cb005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 shutdown_connections 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:29:58.473+0000 7f6cb56aa700 1 -- 172.21.15.118:0/2805788534 wait complete. 2024-02-23T00:29:58.475 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:29:59.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:29:58.472259+0000 mon.smithi090 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.118:0/2805788534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:00.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:29:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:29:59.172990+0000 mgr.smithi090.pxxmlz (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:00.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:00.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:01.173269+0000 mgr.smithi090.pxxmlz (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:02.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/1043991288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 msgr2=0x7f7c080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:02.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/1043991288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c080f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f7bf0009fb0 tx=0x7f7bf0009b30).stop 2024-02-23T00:30:02.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/1043991288 shutdown_connections 2024-02-23T00:30:02.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/1043991288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c080f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/1043991288 >> 172.21.15.118:0/1043991288 conn(0x7f7c080f0b70 msgr2=0x7f7c080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/1043991288 shutdown_connections 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.374+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/1043991288 wait complete. 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0c8c2700 1 Processor -- start 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0c8c2700 1 -- start start 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0c8c2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0c8c2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c08060240 con 0x7f7c080f5410 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:43854/0 (socket says 172.21.15.118:43854) 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.375+0000 7f7c0659c700 1 -- 172.21.15.118:0/2156966505 learned_addr learned my addr 172.21.15.118:0/2156966505 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7c0659c700 1 -- 172.21.15.118:0/2156966505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf0005040 con 0x7f7c080f5410 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7c0659c700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7bf0005010 tx=0x7f7bf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bf0009320 con 0x7f7c080f5410 2024-02-23T00:30:02.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c08060440 con 0x7f7c080f5410 2024-02-23T00:30:02.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bf0013380 con 0x7f7c080f5410 2024-02-23T00:30:02.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bf0014e50 con 0x7f7c080f5410 2024-02-23T00:30:02.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.376+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c08109b90 con 0x7f7c080f5410 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.377+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c08045ad0 con 0x7f7c080f5410 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.377+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bf00095f0 con 0x7f7c080f5410 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.377+0000 7f7bfeffd700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 0x7f7bf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.377+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bf0040f10 con 0x7f7c080f5410 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.377+0000 7f7c05d9b700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 0x7f7bf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:02.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.378+0000 7f7c05d9b700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 0x7f7bf40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7bf80096a0 tx=0x7f7bf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:02.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.381+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bf0014420 con 0x7f7c080f5410 2024-02-23T00:30:02.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.593+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c0810a3a0 con 0x7f7c080f5410 2024-02-23T00:30:02.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.594+0000 7f7bfeffd700 1 -- 172.21.15.118:0/2156966505 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bf0010230 con 0x7f7c080f5410 2024-02-23T00:30:02.595 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:02.596 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 msgr2=0x7f7bf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 0x7f7bf40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7bf80096a0 tx=0x7f7bf8006b40).stop 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 msgr2=0x7f7c081094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7bf0005010 tx=0x7f7bf000b040).stop 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 shutdown_connections 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7bf402ff10 0x7f7bf40323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 --2- 172.21.15.118:0/2156966505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c080f5410 0x7f7c081094e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:02.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 >> 172.21.15.118:0/2156966505 conn(0x7f7c080f0b70 msgr2=0x7f7c080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:02.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 shutdown_connections 2024-02-23T00:30:02.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:02.596+0000 7f7c0c8c2700 1 -- 172.21.15.118:0/2156966505 wait complete. 2024-02-23T00:30:02.599 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:03.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:02.595023+0000 mon.smithi090 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.118:0/2156966505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:04.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:03.173664+0000 mgr.smithi090.pxxmlz (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:04.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:04.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:05.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:05.174092+0000 mgr.smithi090.pxxmlz (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:06.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- 172.21.15.118:0/1601282035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 msgr2=0x7fe25c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 --2- 172.21.15.118:0/1601282035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f4be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fe250009fb0 tx=0x7fe250009b30).stop 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- 172.21.15.118:0/1601282035 shutdown_connections 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 --2- 172.21.15.118:0/1601282035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f4be0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- 172.21.15.118:0/1601282035 >> 172.21.15.118:0/1601282035 conn(0x7fe25c0f02e0 msgr2=0x7fe25c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- 172.21.15.118:0/1601282035 shutdown_connections 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- 172.21.15.118:0/1601282035 wait complete. 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 Processor -- start 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- start start 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.719+0000 7fe2609d0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0f57f0 con 0x7fe25c0f47c0 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe25a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe25a59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:40804/0 (socket says 172.21.15.118:40804) 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe25a59c700 1 -- 172.21.15.118:0/484661627 learned_addr learned my addr 172.21.15.118:0/484661627 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe25a59c700 1 -- 172.21.15.118:0/484661627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250005040 con 0x7fe25c0f47c0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe25a59c700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe2500089f0 tx=0x7fe25000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe250009320 con 0x7fe25c0f47c0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c0f5990 con 0x7fe25c0f47c0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe250013380 con 0x7fe25c0f47c0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25001ed80 con 0x7fe25c0f47c0 2024-02-23T00:30:06.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.720+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c0f5e20 con 0x7fe25c0f47c0 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.721+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe25001e420 con 0x7fe25c0f47c0 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.721+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c059100 con 0x7fe25c0f47c0 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.721+0000 7fe24affd700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 0x7fe244032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.721+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe250040f30 con 0x7fe25c0f47c0 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.721+0000 7fe259d9b700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 0x7fe244032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.722+0000 7fe259d9b700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 0x7fe244032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe24c0096a0 tx=0x7fe24c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:06.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.726+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe25000e2a0 con 0x7fe25c0f47c0 2024-02-23T00:30:06.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.928+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe25c0fc090 con 0x7fe25c0f47c0 2024-02-23T00:30:06.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.929+0000 7fe24affd700 1 -- 172.21.15.118:0/484661627 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe25000e2a0 con 0x7fe25c0f47c0 2024-02-23T00:30:06.932 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:06.932 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 msgr2=0x7fe244032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 0x7fe244032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe24c0096a0 tx=0x7fe24c006b40).stop 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 msgr2=0x7fe25c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe2500089f0 tx=0x7fe25000b040).stop 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 shutdown_connections 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe24402ffb0 0x7fe244032470 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 --2- 172.21.15.118:0/484661627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe25c0f47c0 0x7fe25c0f7370 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 >> 172.21.15.118:0/484661627 conn(0x7fe25c0f02e0 msgr2=0x7fe25c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:06.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 shutdown_connections 2024-02-23T00:30:06.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:06.931+0000 7fe2609d0700 1 -- 172.21.15.118:0/484661627 wait complete. 2024-02-23T00:30:06.934 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:07.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:06.930761+0000 mon.smithi090 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.118:0/484661627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:08.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:07.174476+0000 mgr.smithi090.pxxmlz (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:08.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:08.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:09.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:09.174921+0000 mgr.smithi090.pxxmlz (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:10.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/879745693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 msgr2=0x7f1cf40f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/879745693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf40f4f20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f1ce4009fb0 tx=0x7f1ce4009b30).stop 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/879745693 shutdown_connections 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/879745693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf40f4f20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/879745693 >> 172.21.15.118:0/879745693 conn(0x7f1cf40f0260 msgr2=0x7f1cf40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/879745693 shutdown_connections 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.912+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/879745693 wait complete. 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf8f5d700 1 Processor -- start 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf8f5d700 1 -- start start 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf8f5d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:11.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf8f5d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4106d10 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:40816/0 (socket says 172.21.15.118:40816) 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf259c700 1 -- 172.21.15.118:0/2786890771 learned_addr learned my addr 172.21.15.118:0/2786890771 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.913+0000 7f1cf259c700 1 -- 172.21.15.118:0/2786890771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4005040 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1cf259c700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1ce4005010 tx=0x7f1ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce4009320 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf4106f10 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ce4013380 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce4014e50 con 0x7f1cf40f4b00 2024-02-23T00:30:11.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.914+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf41073a0 con 0x7f1cf40f4b00 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ce40095f0 con 0x7f1cf40f4b00 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf4045ad0 con 0x7f1cf40f4b00 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1ce2ffd700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 0x7f1cdc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ce4040e90 con 0x7f1cf40f4b00 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1cf1d9b700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 0x7f1cdc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.915+0000 7f1cf1d9b700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 0x7f1cdc0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1ce80096a0 tx=0x7f1ce8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:11.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:10.919+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ce4014420 con 0x7f1cf40f4b00 2024-02-23T00:30:11.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.125+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cf40fa710 con 0x7f1cf40f4b00 2024-02-23T00:30:11.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.127+0000 7f1ce2ffd700 1 -- 172.21.15.118:0/2786890771 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ce4010230 con 0x7f1cf40f4b00 2024-02-23T00:30:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:11.129 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:11.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 msgr2=0x7f1cdc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:11.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 0x7f1cdc0323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1ce80096a0 tx=0x7f1ce8006b40).stop 2024-02-23T00:30:11.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 msgr2=0x7f1cf4106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:11.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1ce4005010 tx=0x7f1ce400b040).stop 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 shutdown_connections 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1cdc02ff10 0x7f1cdc0323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 --2- 172.21.15.118:0/2786890771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cf40f4b00 0x7f1cf4106820 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 >> 172.21.15.118:0/2786890771 conn(0x7f1cf40f0260 msgr2=0x7f1cf40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 shutdown_connections 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:11.128+0000 7f1cf8f5d700 1 -- 172.21.15.118:0/2786890771 wait complete. 2024-02-23T00:30:11.131 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:11.127703+0000 mon.smithi090 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.118:0/2786890771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:11.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:11.175294+0000 mgr.smithi090.pxxmlz (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:12.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:12.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:13.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:13.175572+0000 mgr.smithi090.pxxmlz (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:14.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 -- 172.21.15.118:0/4154943854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 msgr2=0x7f48100f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 --2- 172.21.15.118:0/4154943854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48100f4be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f47f8009fb0 tx=0x7f47f8009b30).stop 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 -- 172.21.15.118:0/4154943854 shutdown_connections 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 --2- 172.21.15.118:0/4154943854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48100f4be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 -- 172.21.15.118:0/4154943854 >> 172.21.15.118:0/4154943854 conn(0x7f48100f02e0 msgr2=0x7f48100f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 -- 172.21.15.118:0/4154943854 shutdown_connections 2024-02-23T00:30:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.920+0000 7f4815647700 1 -- 172.21.15.118:0/4154943854 wait complete. 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f4815647700 1 Processor -- start 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f4815647700 1 -- start start 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f4815647700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f4815647700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4810108ec0 con 0x7f48100f47c0 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f480effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f480effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35326/0 (socket says 172.21.15.118:35326) 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f480effd700 1 -- 172.21.15.118:0/3133394520 learned_addr learned my addr 172.21.15.118:0/3133394520 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f480effd700 1 -- 172.21.15.118:0/3133394520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8005040 con 0x7f48100f47c0 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.921+0000 7f480effd700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f47f8005010 tx=0x7f47f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f8009320 con 0x7f48100f47c0 2024-02-23T00:30:14.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48101090c0 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47f8013380 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f8014e50 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4810109550 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.922+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47f80263a0 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.923+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4810045ad0 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.923+0000 7f48077fe700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 0x7f47fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.923+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47f8040ff0 con 0x7f48100f47c0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.923+0000 7f480e7fc700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 0x7f47fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:14.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.923+0000 7f480e7fc700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 0x7f47fc0323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f48000096a0 tx=0x7f4800006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:14.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:14.927+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47f80126f0 con 0x7f48100f47c0 2024-02-23T00:30:15.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.132+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48100f6b00 con 0x7f48100f47c0 2024-02-23T00:30:15.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.133+0000 7f48077fe700 1 -- 172.21.15.118:0/3133394520 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47f800fbf0 con 0x7f48100f47c0 2024-02-23T00:30:15.134 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:15.134 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:15.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 msgr2=0x7f47fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:15.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 0x7f47fc0323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f48000096a0 tx=0x7f4800006b40).stop 2024-02-23T00:30:15.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 msgr2=0x7f48101089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:15.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f47f8005010 tx=0x7f47f800b040).stop 2024-02-23T00:30:15.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 shutdown_connections 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f47fc02ff10 0x7f47fc0323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 --2- 172.21.15.118:0/3133394520 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48100f47c0 0x7f48101089d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 >> 172.21.15.118:0/3133394520 conn(0x7f48100f02e0 msgr2=0x7f48100f1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 shutdown_connections 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:15.135+0000 7f4815647700 1 -- 172.21.15.118:0/3133394520 wait complete. 2024-02-23T00:30:15.137 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:15.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:15.133768+0000 mon.smithi090 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.118:0/3133394520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:16.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:15.175849+0000 mgr.smithi090.pxxmlz (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:16.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:16.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:18.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:17.176195+0000 mgr.smithi090.pxxmlz (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- 172.21.15.118:0/2671226041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 msgr2=0x7f83d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 --2- 172.21.15.118:0/2671226041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d40f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f83c8009fb0 tx=0x7f83c8009b30).stop 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- 172.21.15.118:0/2671226041 shutdown_connections 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 --2- 172.21.15.118:0/2671226041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d40f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- 172.21.15.118:0/2671226041 >> 172.21.15.118:0/2671226041 conn(0x7f83d40f0b70 msgr2=0x7f83d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- 172.21.15.118:0/2671226041 shutdown_connections 2024-02-23T00:30:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- 172.21.15.118:0/2671226041 wait complete. 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 Processor -- start 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.035+0000 7f83dac67700 1 -- start start 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83dac67700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83dac67700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d4106d00 con 0x7f83d40f5410 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d8a03700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d8a03700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35356/0 (socket says 172.21.15.118:35356) 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d8a03700 1 -- 172.21.15.118:0/3934052842 learned_addr learned my addr 172.21.15.118:0/3934052842 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d8a03700 1 -- 172.21.15.118:0/3934052842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c8005040 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d8a03700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f83c8005010 tx=0x7f83c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c8009320 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c8013380 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d4106f00 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c8014e50 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.036+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d4107390 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.037+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83c8014420 con 0x7f83d40f5410 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.037+0000 7f83d17fa700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 0x7f83c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.037+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d4045ad0 con 0x7f83d40f5410 2024-02-23T00:30:19.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.037+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83c8040f70 con 0x7f83d40f5410 2024-02-23T00:30:19.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.038+0000 7f83d3fff700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 0x7f83c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:19.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.038+0000 7f83d3fff700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 0x7f83c40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:19.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.042+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83c80095f0 con 0x7f83d40f5410 2024-02-23T00:30:19.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.248+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83d40fb090 con 0x7f83d40f5410 2024-02-23T00:30:19.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.249+0000 7f83d17fa700 1 -- 172.21.15.118:0/3934052842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83c8015040 con 0x7f83d40f5410 2024-02-23T00:30:19.250 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:19.250 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 msgr2=0x7f83c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 0x7f83c40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).stop 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 msgr2=0x7f83d4106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f83c8005010 tx=0x7f83c800b040).stop 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 shutdown_connections 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f83c402ff10 0x7f83c40323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.250+0000 7f83dac67700 1 --2- 172.21.15.118:0/3934052842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d40f5410 0x7f83d4106810 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.251+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 >> 172.21.15.118:0/3934052842 conn(0x7f83d40f0b70 msgr2=0x7f83d405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.251+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 shutdown_connections 2024-02-23T00:30:19.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:19.251+0000 7f83dac67700 1 -- 172.21.15.118:0/3934052842 wait complete. 2024-02-23T00:30:19.253 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:20.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:19.176607+0000 mgr.smithi090.pxxmlz (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:20.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:19.249574+0000 mon.smithi090 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.118:0/3934052842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:21.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:21.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:21.176916+0000 mgr.smithi090.pxxmlz (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:23.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 -- 172.21.15.118:0/46169633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 msgr2=0x7fd0bc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/46169633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc0f4be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fd0ac009fb0 tx=0x7fd0ac009b30).stop 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 -- 172.21.15.118:0/46169633 shutdown_connections 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/46169633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc0f4be0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 -- 172.21.15.118:0/46169633 >> 172.21.15.118:0/46169633 conn(0x7fd0bc0f02e0 msgr2=0x7fd0bc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 -- 172.21.15.118:0/46169633 shutdown_connections 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.194+0000 7fd0c13e1700 1 -- 172.21.15.118:0/46169633 wait complete. 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0c13e1700 1 Processor -- start 2024-02-23T00:30:23.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0c13e1700 1 -- start start 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0c13e1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0c13e1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc106cc0 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0baffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0baffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35372/0 (socket says 172.21.15.118:35372) 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0baffd700 1 -- 172.21.15.118:0/4047919728 learned_addr learned my addr 172.21.15.118:0/4047919728 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0baffd700 1 -- 172.21.15.118:0/4047919728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac005040 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0baffd700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd0ac005010 tx=0x7fd0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.195+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0ac009320 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc106ec0 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0ac013380 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0ac014e50 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc107350 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0ac014420 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc045ad0 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.196+0000 7fd0a37fe700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 0x7fd0a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.197+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0ac040eb0 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.197+0000 7fd0ba7fc700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 0x7fd0a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.197+0000 7fd0ba7fc700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 0x7fd0a40323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:23.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.201+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0ac0095f0 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.409+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0bc108200 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.409+0000 7fd0a37fe700 1 -- 172.21.15.118:0/4047919728 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0ac01db70 con 0x7fd0bc0f47c0 2024-02-23T00:30:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:23.411 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:23.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 msgr2=0x7fd0a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:23.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 0x7fd0a40323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd0b00096a0 tx=0x7fd0b0006b40).stop 2024-02-23T00:30:23.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 msgr2=0x7fd0bc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd0ac005010 tx=0x7fd0ac00b040).stop 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 shutdown_connections 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd0a402ff10 0x7fd0a40323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 --2- 172.21.15.118:0/4047919728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0bc0f47c0 0x7fd0bc1067d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 >> 172.21.15.118:0/4047919728 conn(0x7fd0bc0f02e0 msgr2=0x7fd0bc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 shutdown_connections 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:23.411+0000 7fd0c13e1700 1 -- 172.21.15.118:0/4047919728 wait complete. 2024-02-23T00:30:23.413 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:23.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:23.177242+0000 mgr.smithi090.pxxmlz (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:24.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:23.410498+0000 mon.smithi090 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.118:0/4047919728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:25.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:25.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:25.177590+0000 mgr.smithi090.pxxmlz (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:25.249177+0000 mon.smithi090 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:30:25.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:25.256308+0000 mon.smithi090 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:30:27.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 -- 172.21.15.118:0/1179195627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 msgr2=0x7fc0c00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/1179195627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c00f4b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc0b4009fb0 tx=0x7fc0b4009b30).stop 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 -- 172.21.15.118:0/1179195627 shutdown_connections 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/1179195627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c00f4b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 -- 172.21.15.118:0/1179195627 >> 172.21.15.118:0/1179195627 conn(0x7fc0c00f0260 msgr2=0x7fc0c00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 -- 172.21.15.118:0/1179195627 shutdown_connections 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.209+0000 7fc0c6af9700 1 -- 172.21.15.118:0/1179195627 wait complete. 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c6af9700 1 Processor -- start 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c6af9700 1 -- start start 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c6af9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c6af9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0106ca0 con 0x7fc0c00f4740 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c4895700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c4895700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41904/0 (socket says 172.21.15.118:41904) 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.210+0000 7fc0c4895700 1 -- 172.21.15.118:0/197213555 learned_addr learned my addr 172.21.15.118:0/197213555 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:27.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0c4895700 1 -- 172.21.15.118:0/197213555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4005040 con 0x7fc0c00f4740 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0c4895700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc0b4005010 tx=0x7fc0b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b4009320 con 0x7fc0c00f4740 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0b4013380 con 0x7fc0c00f4740 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c0106ea0 con 0x7fc0c00f4740 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b4014e50 con 0x7fc0c00f4740 2024-02-23T00:30:27.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.211+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c0107330 con 0x7fc0c00f4740 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.212+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0b40263a0 con 0x7fc0c00f4740 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.212+0000 7fc0bd7fa700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 0x7fc0b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.212+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0b4040e60 con 0x7fc0c00f4740 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.212+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0045ad0 con 0x7fc0c00f4740 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.212+0000 7fc0bffff700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 0x7fc0b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.213+0000 7fc0bffff700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 0x7fc0b00323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc0ac0096a0 tx=0x7fc0ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.216+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0b4012920 con 0x7fc0c00f4740 2024-02-23T00:30:27.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.423+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0c00fc170 con 0x7fc0c00f4740 2024-02-23T00:30:27.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.424+0000 7fc0bd7fa700 1 -- 172.21.15.118:0/197213555 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0b402b020 con 0x7fc0c00f4740 2024-02-23T00:30:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:27.425 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 msgr2=0x7fc0b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 0x7fc0b00323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc0ac0096a0 tx=0x7fc0ac006b40).stop 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 msgr2=0x7fc0c01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc0b4005010 tx=0x7fc0b400b040).stop 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 shutdown_connections 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc0b002ff10 0x7fc0b00323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 --2- 172.21.15.118:0/197213555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0c00f4740 0x7fc0c01067b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 >> 172.21.15.118:0/197213555 conn(0x7fc0c00f0260 msgr2=0x7fc0c00f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 shutdown_connections 2024-02-23T00:30:27.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:27.425+0000 7fc0c6af9700 1 -- 172.21.15.118:0/197213555 wait complete. 2024-02-23T00:30:27.428 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:27.177942+0000 mgr.smithi090.pxxmlz (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:28.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:27.424709+0000 mon.smithi090 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.118:0/197213555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:29.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:29.094 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:29.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:29.178264+0000 mgr.smithi090.pxxmlz (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 -- 172.21.15.118:0/1214243608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 msgr2=0x7f2d280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 --2- 172.21.15.118:0/1214243608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d280f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2d18009fb0 tx=0x7f2d18009b30).stop 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 -- 172.21.15.118:0/1214243608 shutdown_connections 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 --2- 172.21.15.118:0/1214243608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d280f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 -- 172.21.15.118:0/1214243608 >> 172.21.15.118:0/1214243608 conn(0x7f2d280f0b70 msgr2=0x7f2d280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:31.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 -- 172.21.15.118:0/1214243608 shutdown_connections 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 -- 172.21.15.118:0/1214243608 wait complete. 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.127+0000 7f2d302db700 1 Processor -- start 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d302db700 1 -- start start 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d302db700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d302db700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28060280 con 0x7f2d280f5410 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d2e077700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d2e077700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41930/0 (socket says 172.21.15.118:41930) 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d2e077700 1 -- 172.21.15.118:0/1864172614 learned_addr learned my addr 172.21.15.118:0/1864172614 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d2e077700 1 -- 172.21.15.118:0/1864172614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d280f5410 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d2e077700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2d180089f0 tx=0x7f2d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18009320 con 0x7f2d280f5410 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28060480 con 0x7f2d280f5410 2024-02-23T00:30:31.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d18013380 con 0x7f2d280f5410 2024-02-23T00:30:31.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d1801ed80 con 0x7f2d280f5410 2024-02-23T00:30:31.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.128+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28109bd0 con 0x7f2d280f5410 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.129+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d1801e420 con 0x7f2d280f5410 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.129+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d28045ad0 con 0x7f2d280f5410 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.129+0000 7f2d1effd700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.129+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d180410c0 con 0x7f2d280f5410 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.130+0000 7f2d2d876700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:31.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.130+0000 7f2d2d876700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:31.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.134+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d180095f0 con 0x7f2d280f5410 2024-02-23T00:30:31.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.340+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d280fafa0 con 0x7f2d280f5410 2024-02-23T00:30:31.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.340+0000 7f2d1effd700 1 -- 172.21.15.118:0/1864172614 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d1801cb10 con 0x7f2d280f5410 2024-02-23T00:30:31.346 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:31.346 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:31.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 msgr2=0x7f2d14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:31.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).stop 2024-02-23T00:30:31.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 msgr2=0x7f2d281094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:31.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2d180089f0 tx=0x7f2d1800b040).stop 2024-02-23T00:30:31.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 shutdown_connections 2024-02-23T00:30:31.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:31.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 --2- 172.21.15.118:0/1864172614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d280f5410 0x7f2d281094a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:31.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 >> 172.21.15.118:0/1864172614 conn(0x7f2d280f0b70 msgr2=0x7f2d280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:31.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 shutdown_connections 2024-02-23T00:30:31.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:31.342+0000 7f2d302db700 1 -- 172.21.15.118:0/1864172614 wait complete. 2024-02-23T00:30:31.352 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:31.178573+0000 mgr.smithi090.pxxmlz (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:31.341171+0000 mon.smithi090 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.118:0/1864172614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:33.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:33.423 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:33.178891+0000 mgr.smithi090.pxxmlz (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:35.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 -- 172.21.15.118:0/411016504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 msgr2=0x7efc480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 --2- 172.21.15.118:0/411016504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc480f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7efc3c009fb0 tx=0x7efc3c009b30).stop 2024-02-23T00:30:35.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 -- 172.21.15.118:0/411016504 shutdown_connections 2024-02-23T00:30:35.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 --2- 172.21.15.118:0/411016504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc480f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 -- 172.21.15.118:0/411016504 >> 172.21.15.118:0/411016504 conn(0x7efc480f09a0 msgr2=0x7efc480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 -- 172.21.15.118:0/411016504 shutdown_connections 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.462+0000 7efc4d99e700 1 -- 172.21.15.118:0/411016504 wait complete. 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.463+0000 7efc4d99e700 1 Processor -- start 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.463+0000 7efc4d99e700 1 -- start start 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.463+0000 7efc4d99e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.463+0000 7efc4d99e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc48060280 con 0x7efc480f51a0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4c99c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4c99c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59482/0 (socket says 172.21.15.118:59482) 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4c99c700 1 -- 172.21.15.118:0/2527139098 learned_addr learned my addr 172.21.15.118:0/2527139098 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4c99c700 1 -- 172.21.15.118:0/2527139098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c005040 con 0x7efc480f51a0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4c99c700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7efc3c0089f0 tx=0x7efc3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c009320 con 0x7efc480f51a0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc3c013380 con 0x7efc480f51a0 2024-02-23T00:30:35.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc48060480 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c014e20 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.464+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc481099c0 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.465+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc3c0095f0 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.465+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc48045ad0 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.465+0000 7efc457fa700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 0x7efc38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.465+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc3c040f50 con 0x7efc480f51a0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.465+0000 7efc47fff700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 0x7efc38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.466+0000 7efc47fff700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 0x7efc38032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efc340096a0 tx=0x7efc34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.470+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc3c014420 con 0x7efc480f51a0 2024-02-23T00:30:35.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:35.179254+0000 mgr.smithi090.pxxmlz (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.674+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc480fad30 con 0x7efc480f51a0 2024-02-23T00:30:35.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.675+0000 7efc457fa700 1 -- 172.21.15.118:0/2527139098 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc3c01d980 con 0x7efc480f51a0 2024-02-23T00:30:35.677 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:35.677 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:35.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 msgr2=0x7efc38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 0x7efc38032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efc340096a0 tx=0x7efc34006b40).stop 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 msgr2=0x7efc48109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7efc3c0089f0 tx=0x7efc3c00b040).stop 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 shutdown_connections 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7efc3802ff60 0x7efc38032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 --2- 172.21.15.118:0/2527139098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc480f51a0 0x7efc48109290 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 >> 172.21.15.118:0/2527139098 conn(0x7efc480f09a0 msgr2=0x7efc480f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 shutdown_connections 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:35.677+0000 7efc4d99e700 1 -- 172.21.15.118:0/2527139098 wait complete. 2024-02-23T00:30:35.679 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:36.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:35.676180+0000 mon.smithi090 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.118:0/2527139098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:37.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:37.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:37.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:37.179645+0000 mgr.smithi090.pxxmlz (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:39.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 -- 172.21.15.118:0/407609637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 msgr2=0x7f01780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:39.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 --2- 172.21.15.118:0/407609637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01780f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f016c009fb0 tx=0x7f016c009b30).stop 2024-02-23T00:30:39.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 -- 172.21.15.118:0/407609637 shutdown_connections 2024-02-23T00:30:39.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 --2- 172.21.15.118:0/407609637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01780f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:39.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 -- 172.21.15.118:0/407609637 >> 172.21.15.118:0/407609637 conn(0x7f01780f0b70 msgr2=0x7f01780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 -- 172.21.15.118:0/407609637 shutdown_connections 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.489+0000 7f017d67b700 1 -- 172.21.15.118:0/407609637 wait complete. 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f017d67b700 1 Processor -- start 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f017d67b700 1 -- start start 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f017d67b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f017d67b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0178105bd0 con 0x7f01780f5410 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f0176ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f0176ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59494/0 (socket says 172.21.15.118:59494) 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.490+0000 7f0176ffd700 1 -- 172.21.15.118:0/1730760765 learned_addr learned my addr 172.21.15.118:0/1730760765 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f0176ffd700 1 -- 172.21.15.118:0/1730760765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c005040 con 0x7f01780f5410 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f0176ffd700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f016c005010 tx=0x7f016c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016c009320 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f016c013380 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0178105dd0 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016c014e50 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.491+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0178106260 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f016c014420 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0178045ad0 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f015f7fe700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 0x7f01600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f016c040f40 con 0x7f01780f5410 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f01767fc700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 0x7f01600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.492+0000 7f01767fc700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 0x7f01600323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f01680096a0 tx=0x7f0168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:39.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.497+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f016c0095f0 con 0x7f01780f5410 2024-02-23T00:30:39.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:39.179893+0000 mgr.smithi090.pxxmlz (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:39.703 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.702+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0178107110 con 0x7f01780f5410 2024-02-23T00:30:39.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.703+0000 7f015f7fe700 1 -- 172.21.15.118:0/1730760765 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f016c01db70 con 0x7f01780f5410 2024-02-23T00:30:39.704 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:39.705 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:39.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 msgr2=0x7f01600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 0x7f01600323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f01680096a0 tx=0x7f0168006b40).stop 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 msgr2=0x7f01781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f016c005010 tx=0x7f016c00b040).stop 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 shutdown_connections 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f016002ff10 0x7f01600323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 --2- 172.21.15.118:0/1730760765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780f5410 0x7f01781056e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 >> 172.21.15.118:0/1730760765 conn(0x7f01780f0b70 msgr2=0x7f01780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 shutdown_connections 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:39.705+0000 7f017d67b700 1 -- 172.21.15.118:0/1730760765 wait complete. 2024-02-23T00:30:39.707 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:40.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:39.703728+0000 mon.smithi090 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.118:0/1730760765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:41.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:41.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:41.180162+0000 mgr.smithi090.pxxmlz (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:43.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:43.180514+0000 mgr.smithi090.pxxmlz (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 -- 172.21.15.118:0/942446943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 msgr2=0x7f4c440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/942446943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c440f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f4c2c009fb0 tx=0x7f4c2c009b30).stop 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 -- 172.21.15.118:0/942446943 shutdown_connections 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/942446943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c440f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 -- 172.21.15.118:0/942446943 >> 172.21.15.118:0/942446943 conn(0x7f4c440f0b70 msgr2=0x7f4c440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 -- 172.21.15.118:0/942446943 shutdown_connections 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.782+0000 7f4c48a3c700 1 -- 172.21.15.118:0/942446943 wait complete. 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c48a3c700 1 Processor -- start 2024-02-23T00:30:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c48a3c700 1 -- start start 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c48a3c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c48a3c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c44060240 con 0x7f4c440f5410 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c4259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c4259c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59516/0 (socket says 172.21.15.118:59516) 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.783+0000 7f4c4259c700 1 -- 172.21.15.118:0/3234171522 learned_addr learned my addr 172.21.15.118:0/3234171522 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c4259c700 1 -- 172.21.15.118:0/3234171522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c005040 con 0x7f4c440f5410 2024-02-23T00:30:43.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c4259c700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4c2c005010 tx=0x7f4c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c2c009320 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c44060440 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c2c013380 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c2c014e50 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.784+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c44109b90 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.785+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c2c0263a0 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.785+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c44045ad0 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.785+0000 7f4c3affd700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.785+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c2c041150 con 0x7f4c440f5410 2024-02-23T00:30:43.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.785+0000 7f4c41d9b700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.786+0000 7f4c41d9b700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 0x7f4c300323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4c340096a0 tx=0x7f4c34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:43.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.790+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c2c0126f0 con 0x7f4c440f5410 2024-02-23T00:30:43.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.998+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c4410a3b0 con 0x7f4c440f5410 2024-02-23T00:30:44.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:43.998+0000 7f4c3affd700 1 -- 172.21.15.118:0/3234171522 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c2c012330 con 0x7f4c440f5410 2024-02-23T00:30:44.000 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:44.000 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 msgr2=0x7f4c300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 0x7f4c300323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4c340096a0 tx=0x7f4c34006b40).stop 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 msgr2=0x7f4c441094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4c2c005010 tx=0x7f4c2c00b040).stop 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 shutdown_connections 2024-02-23T00:30:44.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:44.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 --2- 172.21.15.118:0/3234171522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c440f5410 0x7f4c441094e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:44.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 >> 172.21.15.118:0/3234171522 conn(0x7f4c440f0b70 msgr2=0x7f4c440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:44.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 shutdown_connections 2024-02-23T00:30:44.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:44.000+0000 7f4c48a3c700 1 -- 172.21.15.118:0/3234171522 wait complete. 2024-02-23T00:30:44.002 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:44.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:43.999384+0000 mon.smithi090 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.118:0/3234171522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:45.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:45.180805+0000 mgr.smithi090.pxxmlz (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:45.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:45.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:47.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:47.181139+0000 mgr.smithi090.pxxmlz (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- 172.21.15.118:0/2760356541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 msgr2=0x7f89cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 --2- 172.21.15.118:0/2760356541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc0f66c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f89c0009fb0 tx=0x7f89c0009b30).stop 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- 172.21.15.118:0/2760356541 shutdown_connections 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 --2- 172.21.15.118:0/2760356541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc0f66c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- 172.21.15.118:0/2760356541 >> 172.21.15.118:0/2760356541 conn(0x7f89cc0f1aa0 msgr2=0x7f89cc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- 172.21.15.118:0/2760356541 shutdown_connections 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- 172.21.15.118:0/2760356541 wait complete. 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 Processor -- start 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- start start 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.819+0000 7f89cb59e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc106ac0 con 0x7f89cc0f62a0 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89ca59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89ca59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47626/0 (socket says 172.21.15.118:47626) 2024-02-23T00:30:47.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89ca59c700 1 -- 172.21.15.118:0/1915148594 learned_addr learned my addr 172.21.15.118:0/1915148594 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89ca59c700 1 -- 172.21.15.118:0/1915148594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0005040 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89ca59c700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f89c00089f0 tx=0x7f89c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c0009320 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89c0013380 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc106cc0 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c0014e20 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.820+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc107150 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.821+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89c00263a0 con 0x7f89cc0f62a0 2024-02-23T00:30:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.821+0000 7f89baffd700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 0x7f89b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.821+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89c0040e80 con 0x7f89cc0f62a0 2024-02-23T00:30:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.821+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc045ad0 con 0x7f89cc0f62a0 2024-02-23T00:30:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.821+0000 7f89c9d9b700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 0x7f89b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.822+0000 7f89c9d9b700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 0x7f89b4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f89bc0096a0 tx=0x7f89bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:47.826+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89c0012920 con 0x7f89cc0f62a0 2024-02-23T00:30:48.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.031+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89cc108000 con 0x7f89cc0f62a0 2024-02-23T00:30:48.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.031+0000 7f89baffd700 1 -- 172.21.15.118:0/1915148594 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89c002b020 con 0x7f89cc0f62a0 2024-02-23T00:30:48.033 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:48.033 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 msgr2=0x7f89b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 0x7f89b4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f89bc0096a0 tx=0x7f89bc006b40).stop 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 msgr2=0x7f89cc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f89c00089f0 tx=0x7f89c000b040).stop 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 shutdown_connections 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f89b402ff60 0x7f89b4032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 --2- 172.21.15.118:0/1915148594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89cc0f62a0 0x7f89cc1065d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 >> 172.21.15.118:0/1915148594 conn(0x7f89cc0f1aa0 msgr2=0x7f89cc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 shutdown_connections 2024-02-23T00:30:48.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:48.033+0000 7f89cb59e700 1 -- 172.21.15.118:0/1915148594 wait complete. 2024-02-23T00:30:48.036 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:48.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:48 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:48.032384+0000 mon.smithi090 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.118:0/1915148594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:49.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:49.181386+0000 mgr.smithi090.pxxmlz (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:49.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:49.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:51.181665+0000 mgr.smithi090.pxxmlz (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 -- 172.21.15.118:0/1640685067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 msgr2=0x7f26600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 --2- 172.21.15.118:0/1640685067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26600f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2654009fb0 tx=0x7f2654009b30).stop 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 -- 172.21.15.118:0/1640685067 shutdown_connections 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 --2- 172.21.15.118:0/1640685067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26600f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 -- 172.21.15.118:0/1640685067 >> 172.21.15.118:0/1640685067 conn(0x7f26600f1c70 msgr2=0x7f26600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 -- 172.21.15.118:0/1640685067 shutdown_connections 2024-02-23T00:30:52.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 -- 172.21.15.118:0/1640685067 wait complete. 2024-02-23T00:30:52.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.029+0000 7f26661cf700 1 Processor -- start 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f26661cf700 1 -- start start 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f26661cf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f26661cf700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2660106cd0 con 0x7f26600f6510 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f265f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47638/0 (socket says 172.21.15.118:47638) 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f265f7fe700 1 -- 172.21.15.118:0/2442725602 learned_addr learned my addr 172.21.15.118:0/2442725602 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f265f7fe700 1 -- 172.21.15.118:0/2442725602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654005040 con 0x7f26600f6510 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f265f7fe700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2654005010 tx=0x7f265400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.030+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654009320 con 0x7f26600f6510 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2660106ed0 con 0x7f26600f6510 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2654013380 con 0x7f26600f6510 2024-02-23T00:30:52.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654014e50 con 0x7f26600f6510 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660107360 con 0x7f26600f6510 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26540263a0 con 0x7f26600f6510 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.031+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660045ad0 con 0x7f26600f6510 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.032+0000 7f2647fff700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 0x7f26480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.032+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2654041150 con 0x7f26600f6510 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.032+0000 7f265effd700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 0x7f26480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.032+0000 7f265effd700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 0x7f26480323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f26500096a0 tx=0x7f2650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:52.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.036+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f265401ccf0 con 0x7f26600f6510 2024-02-23T00:30:52.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.244+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2660108210 con 0x7f26600f6510 2024-02-23T00:30:52.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.245+0000 7f2647fff700 1 -- 172.21.15.118:0/2442725602 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f265401cb10 con 0x7f26600f6510 2024-02-23T00:30:52.256 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:52.256 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:52.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 msgr2=0x7f26480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 0x7f26480323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f26500096a0 tx=0x7f2650006b40).stop 2024-02-23T00:30:52.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 msgr2=0x7f26601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2654005010 tx=0x7f265400b040).stop 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 shutdown_connections 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f264802ff10 0x7f26480323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 --2- 172.21.15.118:0/2442725602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26600f6510 0x7f26601067e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 >> 172.21.15.118:0/2442725602 conn(0x7f26600f1c70 msgr2=0x7f26600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 shutdown_connections 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:52.247+0000 7f26661cf700 1 -- 172.21.15.118:0/2442725602 wait complete. 2024-02-23T00:30:52.258 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:52.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:52 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:52.247122+0000 mon.smithi090 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.118:0/2442725602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:53.181969+0000 mgr.smithi090.pxxmlz (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:53.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:53.963 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:55.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:55.182300+0000 mgr.smithi090.pxxmlz (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 -- 172.21.15.118:0/1945896212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 msgr2=0x7f77a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 --2- 172.21.15.118:0/1945896212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f7798009fb0 tx=0x7f7798009b30).stop 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 -- 172.21.15.118:0/1945896212 shutdown_connections 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 --2- 172.21.15.118:0/1945896212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 -- 172.21.15.118:0/1945896212 >> 172.21.15.118:0/1945896212 conn(0x7f77a80f0b70 msgr2=0x7f77a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 -- 172.21.15.118:0/1945896212 shutdown_connections 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.023+0000 7f77affd3700 1 -- 172.21.15.118:0/1945896212 wait complete. 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77affd3700 1 Processor -- start 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77affd3700 1 -- start start 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77affd3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77affd3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80fb960 con 0x7f77a80f5410 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77add6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77add6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39652/0 (socket says 172.21.15.118:39652) 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77add6f700 1 -- 172.21.15.118:0/3538795722 learned_addr learned my addr 172.21.15.118:0/3538795722 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77add6f700 1 -- 172.21.15.118:0/3538795722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798005040 con 0x7f77a80f5410 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.024+0000 7f77add6f700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7798005010 tx=0x7f779800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7798009320 con 0x7f77a80f5410 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80fbb00 con 0x7f77a80f5410 2024-02-23T00:30:56.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7798013380 con 0x7f77a80f5410 2024-02-23T00:30:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7798014e50 con 0x7f77a80f5410 2024-02-23T00:30:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a80fbf90 con 0x7f77a80f5410 2024-02-23T00:30:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7798014420 con 0x7f77a80f5410 2024-02-23T00:30:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.025+0000 7f779e7fc700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.026+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a8059100 con 0x7f77a80f5410 2024-02-23T00:30:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.026+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7798040e30 con 0x7f77a80f5410 2024-02-23T00:30:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.026+0000 7f77ad56e700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.026+0000 7f77ad56e700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 0x7f77940323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f77a40096a0 tx=0x7f77a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:56.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.031+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77980095f0 con 0x7f77a80f5410 2024-02-23T00:30:56.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.235+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77a80fafa0 con 0x7f77a80f5410 2024-02-23T00:30:56.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.236+0000 7f779e7fc700 1 -- 172.21.15.118:0/3538795722 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f779802b020 con 0x7f77a80f5410 2024-02-23T00:30:56.239 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:30:56.239 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:30:56.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 msgr2=0x7f77940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:56.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 0x7f77940323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f77a40096a0 tx=0x7f77a4006b40).stop 2024-02-23T00:30:56.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 msgr2=0x7f77a80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:56.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f7798005010 tx=0x7f779800b040).stop 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 shutdown_connections 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 --2- 172.21.15.118:0/3538795722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77a80f5410 0x7f77a80fd4e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 >> 172.21.15.118:0/3538795722 conn(0x7f77a80f0b70 msgr2=0x7f77a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 shutdown_connections 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:56.239+0000 7f77affd3700 1 -- 172.21.15.118:0/3538795722 wait complete. 2024-02-23T00:30:56.241 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:30:56.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:30:55.310388+0000 mgr.smithi090.pxxmlz (mgr.14162) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi090 ... 2024-02-23T00:30:56.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Verifying port 9100 ... 2024-02-23T00:30:56.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 1 from systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090 2024-02-23T00:30:56.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: systemctl: stderr Job for ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090.service failed because a timeout was exceeded. 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: systemctl: stderr See "systemctl status ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090.service" and "journalctl -xe" for details. 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ports=daemon_ports) 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-23T00:30:56.774 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: Failed command: systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: yield (conn, connr) 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: code, '\n'.join(err))) 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi090 ... 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Verifying port 9100 ... 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 1 from systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090 2024-02-23T00:30:56.775 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: systemctl: stderr Job for ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090.service failed because a timeout was exceeded. 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: systemctl: stderr See "systemctl status ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090.service" and "journalctl -xe" for details. 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: ports=daemon_ports) 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-23T00:30:56.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: Failed command: systemctl start ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@node-exporter.smithi090 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:30:55.315876+0000 mgr.smithi090.pxxmlz (mgr.14162) 233 : cephadm [INF] Deploying daemon prometheus.smithi090 on smithi090 2024-02-23T00:30:56.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:30:56.238113+0000 mon.smithi090 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.118:0/3538795722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:30:57.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:57.182613+0000 mgr.smithi090.pxxmlz (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:57.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:30:57.913 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:30:59.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:30:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:30:59.182904+0000 mgr.smithi090.pxxmlz (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 -- 172.21.15.118:0/3433074923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 msgr2=0x7f7b340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 --2- 172.21.15.118:0/3433074923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b340f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7b24009fb0 tx=0x7f7b24009b30).stop 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 -- 172.21.15.118:0/3433074923 shutdown_connections 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 --2- 172.21.15.118:0/3433074923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b340f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 -- 172.21.15.118:0/3433074923 >> 172.21.15.118:0/3433074923 conn(0x7f7b340f0b70 msgr2=0x7f7b340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 -- 172.21.15.118:0/3433074923 shutdown_connections 2024-02-23T00:30:59.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.946+0000 7f7b39be9700 1 -- 172.21.15.118:0/3433074923 wait complete. 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.947+0000 7f7b39be9700 1 Processor -- start 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.947+0000 7f7b39be9700 1 -- start start 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.947+0000 7f7b39be9700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.947+0000 7f7b39be9700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b34106cf0 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.947+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b337fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39662/0 (socket says 172.21.15.118:39662) 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b337fe700 1 -- 172.21.15.118:0/911900195 learned_addr learned my addr 172.21.15.118:0/911900195 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b337fe700 1 -- 172.21.15.118:0/911900195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24005040 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b337fe700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f7b240089f0 tx=0x7f7b2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b24009320 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b34106ef0 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b24013380 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b2401ed80 con 0x7f7b340f5410 2024-02-23T00:30:59.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.948+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b34107380 con 0x7f7b340f5410 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.949+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b34045ad0 con 0x7f7b340f5410 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.949+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b2401e420 con 0x7f7b340f5410 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.949+0000 7f7b2bfff700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.949+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b24040ff0 con 0x7f7b340f5410 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.949+0000 7f7b32ffd700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.950+0000 7f7b32ffd700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 0x7f7b20032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7b1c0096a0 tx=0x7f7b1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:59.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:30:59.954+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b240095f0 con 0x7f7b340f5410 2024-02-23T00:31:00.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.162+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b341081d0 con 0x7f7b340f5410 2024-02-23T00:31:00.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.163+0000 7f7b2bfff700 1 -- 172.21.15.118:0/911900195 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b2401cb10 con 0x7f7b340f5410 2024-02-23T00:31:00.164 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:00.165 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 msgr2=0x7f7b20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 0x7f7b20032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7b1c0096a0 tx=0x7f7b1c006b40).stop 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 msgr2=0x7f7b34106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f7b240089f0 tx=0x7f7b2400b040).stop 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 shutdown_connections 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 --2- 172.21.15.118:0/911900195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b340f5410 0x7f7b34106800 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.164+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 >> 172.21.15.118:0/911900195 conn(0x7f7b340f0b70 msgr2=0x7f7b340f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.165+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 shutdown_connections 2024-02-23T00:31:00.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:00.165+0000 7f7b39be9700 1 -- 172.21.15.118:0/911900195 wait complete. 2024-02-23T00:31:00.167 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:00.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:00.163722+0000 mon.smithi090 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.118:0/911900195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:01.183227+0000 mgr.smithi090.pxxmlz (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:01.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:01.896 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:03.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:03.183560+0000 mgr.smithi090.pxxmlz (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:04.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.179+0000 7f193d65d700 1 -- 172.21.15.118:0/160445416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 msgr2=0x7f19380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:04.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.179+0000 7f193d65d700 1 --2- 172.21.15.118:0/160445416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19380f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f1920009fb0 tx=0x7f1920009b30).stop 2024-02-23T00:31:04.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- 172.21.15.118:0/160445416 shutdown_connections 2024-02-23T00:31:04.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 --2- 172.21.15.118:0/160445416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19380f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:04.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- 172.21.15.118:0/160445416 >> 172.21.15.118:0/160445416 conn(0x7f19380f1c70 msgr2=0x7f19380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:04.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- 172.21.15.118:0/160445416 shutdown_connections 2024-02-23T00:31:04.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- 172.21.15.118:0/160445416 wait complete. 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 Processor -- start 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- start start 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.180+0000 7f193d65d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1938106cd0 con 0x7f19380f6510 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f1936ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f1936ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39684/0 (socket says 172.21.15.118:39684) 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f1936ffd700 1 -- 172.21.15.118:0/1016553328 learned_addr learned my addr 172.21.15.118:0/1016553328 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f1936ffd700 1 -- 172.21.15.118:0/1016553328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1920005040 con 0x7f19380f6510 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f1936ffd700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f1920005010 tx=0x7f192000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:04.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1920009320 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1938106ed0 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1920013380 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1920014e50 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.181+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1938107360 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.182+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19200095f0 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.182+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1938045ad0 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.182+0000 7f192f7fe700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 0x7f19240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.182+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1920040fa0 con 0x7f19380f6510 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.183+0000 7f19367fc700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 0x7f19240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:04.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.183+0000 7f19367fc700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 0x7f19240323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f19280096a0 tx=0x7f1928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:04.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.187+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1920014420 con 0x7f19380f6510 2024-02-23T00:31:04.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.391+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1938108210 con 0x7f19380f6510 2024-02-23T00:31:04.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.392+0000 7f192f7fe700 1 -- 172.21.15.118:0/1016553328 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1920010230 con 0x7f19380f6510 2024-02-23T00:31:04.393 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:04.393 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 msgr2=0x7f19240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 0x7f19240323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f19280096a0 tx=0x7f1928006b40).stop 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 msgr2=0x7f19381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f1920005010 tx=0x7f192000b040).stop 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 shutdown_connections 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f192402ff10 0x7f19240323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 --2- 172.21.15.118:0/1016553328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19380f6510 0x7f19381067e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 >> 172.21.15.118:0/1016553328 conn(0x7f19380f1c70 msgr2=0x7f19380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 shutdown_connections 2024-02-23T00:31:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:04.393+0000 7f193d65d700 1 -- 172.21.15.118:0/1016553328 wait complete. 2024-02-23T00:31:04.396 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:04.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:04.392607+0000 mon.smithi090 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.118:0/1016553328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:05.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:05.183863+0000 mgr.smithi090.pxxmlz (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:06.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:06.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:07.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:07.184186+0000 mgr.smithi090.pxxmlz (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:08.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 -- 172.21.15.118:0/2849358539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 msgr2=0x7f901c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:08.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 --2- 172.21.15.118:0/2849358539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c0f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f900c009fb0 tx=0x7f900c009b30).stop 2024-02-23T00:31:08.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 -- 172.21.15.118:0/2849358539 shutdown_connections 2024-02-23T00:31:08.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 --2- 172.21.15.118:0/2849358539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c0f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:08.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 -- 172.21.15.118:0/2849358539 >> 172.21.15.118:0/2849358539 conn(0x7f901c0f0b70 msgr2=0x7f901c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:08.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 -- 172.21.15.118:0/2849358539 shutdown_connections 2024-02-23T00:31:08.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.131+0000 7f90220b2700 1 -- 172.21.15.118:0/2849358539 wait complete. 2024-02-23T00:31:08.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.132+0000 7f90220b2700 1 Processor -- start 2024-02-23T00:31:08.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.132+0000 7f90220b2700 1 -- start start 2024-02-23T00:31:08.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f90220b2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:08.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f90220b2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c060220 con 0x7f901c0f5410 2024-02-23T00:31:08.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f901b7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:32950/0 (socket says 172.21.15.118:32950) 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f901b7fe700 1 -- 172.21.15.118:0/712512837 learned_addr learned my addr 172.21.15.118:0/712512837 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f901b7fe700 1 -- 172.21.15.118:0/712512837 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c005040 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f901b7fe700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f900c005010 tx=0x7f900c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f900c009320 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c060420 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f900c013380 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f900c014e50 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.133+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c109bb0 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.134+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f900c0263a0 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.134+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc004fa0 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.134+0000 7f9003fff700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 0x7f90040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.134+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f900c040f00 con 0x7f901c0f5410 2024-02-23T00:31:08.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.135+0000 7f901affd700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 0x7f90040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:08.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.135+0000 7f901affd700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 0x7f90040323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f90100096a0 tx=0x7f9010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:08.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.139+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f900c0126f0 con 0x7f901c0f5410 2024-02-23T00:31:08.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.345+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ffc005670 con 0x7f901c0f5410 2024-02-23T00:31:08.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.346+0000 7f9003fff700 1 -- 172.21.15.118:0/712512837 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f900c012330 con 0x7f901c0f5410 2024-02-23T00:31:08.348 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:08.348 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 msgr2=0x7f90040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 0x7f90040323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f90100096a0 tx=0x7f9010006b40).stop 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 msgr2=0x7f901c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f900c005010 tx=0x7f900c00b040).stop 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 shutdown_connections 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f900402ff10 0x7f90040323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:08.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 --2- 172.21.15.118:0/712512837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f901c0f5410 0x7f901c109500 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:08.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 >> 172.21.15.118:0/712512837 conn(0x7f901c0f0b70 msgr2=0x7f901c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:08.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 shutdown_connections 2024-02-23T00:31:08.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:08.348+0000 7f90220b2700 1 -- 172.21.15.118:0/712512837 wait complete. 2024-02-23T00:31:08.350 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:08.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:08.347033+0000 mon.smithi090 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.118:0/712512837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:09.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:09.184438+0000 mgr.smithi090.pxxmlz (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:09.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:09.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:11.184788+0000 mgr.smithi090.pxxmlz (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:12.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 -- 172.21.15.118:0/3148262675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 msgr2=0x7fe1840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 --2- 172.21.15.118:0/3148262675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe16c009fb0 tx=0x7fe16c009b30).stop 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 -- 172.21.15.118:0/3148262675 shutdown_connections 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 --2- 172.21.15.118:0/3148262675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 -- 172.21.15.118:0/3148262675 >> 172.21.15.118:0/3148262675 conn(0x7fe1840f0b70 msgr2=0x7fe1840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 -- 172.21.15.118:0/3148262675 shutdown_connections 2024-02-23T00:31:12.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.098+0000 7fe1890d3700 1 -- 172.21.15.118:0/3148262675 wait complete. 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe1890d3700 1 Processor -- start 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe1890d3700 1 -- start start 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe1890d3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe1890d3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840fb960 con 0x7fe1840f5410 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe182d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe182d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:32980/0 (socket says 172.21.15.118:32980) 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.099+0000 7fe182d9d700 1 -- 172.21.15.118:0/1259098958 learned_addr learned my addr 172.21.15.118:0/1259098958 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe182d9d700 1 -- 172.21.15.118:0/1259098958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c005040 con 0x7fe1840f5410 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe182d9d700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe16c005010 tx=0x7fe16c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe16c009320 con 0x7fe1840f5410 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1840fbb00 con 0x7fe1840f5410 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe16c013380 con 0x7fe1840f5410 2024-02-23T00:31:12.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe16c014e50 con 0x7fe1840f5410 2024-02-23T00:31:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.100+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1840fbf90 con 0x7fe1840f5410 2024-02-23T00:31:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe16c014420 con 0x7fe1840f5410 2024-02-23T00:31:12.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe17b7fe700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 0x7fe1700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe184059100 con 0x7fe1840f5410 2024-02-23T00:31:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe16c040e60 con 0x7fe1840f5410 2024-02-23T00:31:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe18259c700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 0x7fe1700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.101+0000 7fe18259c700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 0x7fe1700323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe1740096a0 tx=0x7fe174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.106+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe16c0095f0 con 0x7fe1840f5410 2024-02-23T00:31:12.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.314+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1840fafa0 con 0x7fe1840f5410 2024-02-23T00:31:12.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.315+0000 7fe17b7fe700 1 -- 172.21.15.118:0/1259098958 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe16c01db70 con 0x7fe1840f5410 2024-02-23T00:31:12.316 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:12.316 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:12.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 msgr2=0x7fe1700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 0x7fe1700323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe1740096a0 tx=0x7fe174006b40).stop 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 msgr2=0x7fe1840fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe16c005010 tx=0x7fe16c00b040).stop 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 shutdown_connections 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe17002ff10 0x7fe1700323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 --2- 172.21.15.118:0/1259098958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1840f5410 0x7fe1840fd4e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 >> 172.21.15.118:0/1259098958 conn(0x7fe1840f0b70 msgr2=0x7fe18405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 shutdown_connections 2024-02-23T00:31:12.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:12.316+0000 7fe1890d3700 1 -- 172.21.15.118:0/1259098958 wait complete. 2024-02-23T00:31:12.319 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:12.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:12.315563+0000 mon.smithi090 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.118:0/1259098958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:13.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:13.185203+0000 mgr.smithi090.pxxmlz (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:14.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:14.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:15.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:15.185576+0000 mgr.smithi090.pxxmlz (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:16.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 -- 172.21.15.118:0/959251401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 msgr2=0x7f2c600f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:16.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 --2- 172.21.15.118:0/959251401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c600f4b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2c54009fb0 tx=0x7f2c54009b30).stop 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 -- 172.21.15.118:0/959251401 shutdown_connections 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 --2- 172.21.15.118:0/959251401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c600f4b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 -- 172.21.15.118:0/959251401 >> 172.21.15.118:0/959251401 conn(0x7f2c600f0260 msgr2=0x7f2c600f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 -- 172.21.15.118:0/959251401 shutdown_connections 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.208+0000 7f2c668a3700 1 -- 172.21.15.118:0/959251401 wait complete. 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c668a3700 1 Processor -- start 2024-02-23T00:31:16.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c668a3700 1 -- start start 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c668a3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c668a3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c60106ca0 con 0x7f2c600f4740 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c5ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c5ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:45042/0 (socket says 172.21.15.118:45042) 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.209+0000 7f2c5ffff700 1 -- 172.21.15.118:0/3499120097 learned_addr learned my addr 172.21.15.118:0/3499120097 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c5ffff700 1 -- 172.21.15.118:0/3499120097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54005040 con 0x7f2c600f4740 2024-02-23T00:31:16.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c5ffff700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2c540089f0 tx=0x7f2c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:16.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c54009320 con 0x7f2c600f4740 2024-02-23T00:31:16.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c60106ea0 con 0x7f2c600f4740 2024-02-23T00:31:16.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c54013380 con 0x7f2c600f4740 2024-02-23T00:31:16.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c5401ed80 con 0x7f2c600f4740 2024-02-23T00:31:16.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.210+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60107330 con 0x7f2c600f4740 2024-02-23T00:31:16.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c54014420 con 0x7f2c600f4740 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c5cff9700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 0x7f2c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c54040fa0 con 0x7f2c600f4740 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c60045ad0 con 0x7f2c600f4740 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c5f7fe700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 0x7f2c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.211+0000 7f2c5f7fe700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 0x7f2c48032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:16.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.216+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c5401c910 con 0x7f2c600f4740 2024-02-23T00:31:16.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.420+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c600fc080 con 0x7f2c600f4740 2024-02-23T00:31:16.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.421+0000 7f2c5cff9700 1 -- 172.21.15.118:0/3499120097 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c5401c320 con 0x7f2c600f4740 2024-02-23T00:31:16.422 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:16.422 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:16.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 msgr2=0x7f2c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:16.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 0x7f2c48032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).stop 2024-02-23T00:31:16.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 msgr2=0x7f2c601067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:16.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2c540089f0 tx=0x7f2c5400b040).stop 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 shutdown_connections 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2c4802ff60 0x7f2c48032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 --2- 172.21.15.118:0/3499120097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c600f4740 0x7f2c601067b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.422+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 >> 172.21.15.118:0/3499120097 conn(0x7f2c600f0260 msgr2=0x7f2c600f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.423+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 shutdown_connections 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:16.423+0000 7f2c668a3700 1 -- 172.21.15.118:0/3499120097 wait complete. 2024-02-23T00:31:16.425 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:16.776 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:16.421860+0000 mon.smithi090 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.118:0/3499120097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:17.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:17.185927+0000 mgr.smithi090.pxxmlz (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:18.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:18.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:19.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:19.186321+0000 mgr.smithi090.pxxmlz (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:20.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 -- 172.21.15.118:0/2884577565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 msgr2=0x7f23500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 --2- 172.21.15.118:0/2884577565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23500f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2340009fb0 tx=0x7f2340009b30).stop 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 -- 172.21.15.118:0/2884577565 shutdown_connections 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 --2- 172.21.15.118:0/2884577565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23500f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 -- 172.21.15.118:0/2884577565 >> 172.21.15.118:0/2884577565 conn(0x7f23500f0b70 msgr2=0x7f23500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 -- 172.21.15.118:0/2884577565 shutdown_connections 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.357+0000 7f23560fc700 1 -- 172.21.15.118:0/2884577565 wait complete. 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f23560fc700 1 Processor -- start 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f23560fc700 1 -- start start 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f23560fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f23560fc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2350106cc0 con 0x7f23500f5410 2024-02-23T00:31:20.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f234f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f234f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:45064/0 (socket says 172.21.15.118:45064) 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.358+0000 7f234f7fe700 1 -- 172.21.15.118:0/186253112 learned_addr learned my addr 172.21.15.118:0/186253112 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f234f7fe700 1 -- 172.21.15.118:0/186253112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340005040 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f234f7fe700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2340005010 tx=0x7f234000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2340009320 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2350106ec0 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2340013380 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2340014e50 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.359+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2350107350 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.360+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23400095f0 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.360+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2350045ad0 con 0x7f23500f5410 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.360+0000 7f2337fff700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 0x7f23380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:20.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.360+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2340040e90 con 0x7f23500f5410 2024-02-23T00:31:20.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.360+0000 7f234effd700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 0x7f23380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:20.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.361+0000 7f234effd700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 0x7f23380323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f23440096a0 tx=0x7f2344006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:20.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.365+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2340014420 con 0x7f23500f5410 2024-02-23T00:31:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.572+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2350108200 con 0x7f23500f5410 2024-02-23T00:31:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.573+0000 7f2337fff700 1 -- 172.21.15.118:0/186253112 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2340010230 con 0x7f23500f5410 2024-02-23T00:31:20.575 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:20.575 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:20.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 msgr2=0x7f23380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:20.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 0x7f23380323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f23440096a0 tx=0x7f2344006b40).stop 2024-02-23T00:31:20.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 msgr2=0x7f23501067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:20.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2340005010 tx=0x7f234000b040).stop 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 shutdown_connections 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f233802ff10 0x7f23380323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 --2- 172.21.15.118:0/186253112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23500f5410 0x7f23501067d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 >> 172.21.15.118:0/186253112 conn(0x7f23500f0b70 msgr2=0x7f235005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 shutdown_connections 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:20.575+0000 7f23560fc700 1 -- 172.21.15.118:0/186253112 wait complete. 2024-02-23T00:31:20.577 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:20.574156+0000 mon.smithi090 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.118:0/186253112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:21.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:21.186618+0000 mgr.smithi090.pxxmlz (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:22.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:22.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:23.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:23.187000+0000 mgr.smithi090.pxxmlz (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- 172.21.15.118:0/315030973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 msgr2=0x7f1eb80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 --2- 172.21.15.118:0/315030973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f8f20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f1ea8009fb0 tx=0x7f1ea8009b30).stop 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- 172.21.15.118:0/315030973 shutdown_connections 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 --2- 172.21.15.118:0/315030973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f8f20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- 172.21.15.118:0/315030973 >> 172.21.15.118:0/315030973 conn(0x7f1eb80f09a0 msgr2=0x7f1eb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- 172.21.15.118:0/315030973 shutdown_connections 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- 172.21.15.118:0/315030973 wait complete. 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 Processor -- start 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- start start 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:24.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.403+0000 7f1ebe283700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80f4e20 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebd281700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebd281700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54690/0 (socket says 172.21.15.118:54690) 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebd281700 1 -- 172.21.15.118:0/1206534728 learned_addr learned my addr 172.21.15.118:0/1206534728 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebd281700 1 -- 172.21.15.118:0/1206534728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebd281700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f1ea8005010 tx=0x7f1ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8009320 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb80f4fc0 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ea8013380 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8014e50 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.404+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb80f5450 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.405+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ea80263a0 con 0x7f1eb80601e0 2024-02-23T00:31:24.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.405+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8059100 con 0x7f1eb80601e0 2024-02-23T00:31:24.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.405+0000 7f1eadffb700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 0x7f1ea40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:24.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.405+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ea8041260 con 0x7f1eb80601e0 2024-02-23T00:31:24.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.405+0000 7f1ebca80700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 0x7f1ea40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:24.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.406+0000 7f1ebca80700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 0x7f1ea40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f1eb40096a0 tx=0x7f1eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:24.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.410+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ea80126f0 con 0x7f1eb80601e0 2024-02-23T00:31:24.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.618+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1eb80fbe10 con 0x7f1eb80601e0 2024-02-23T00:31:24.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.619+0000 7f1eadffb700 1 -- 172.21.15.118:0/1206534728 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ea8012330 con 0x7f1eb80601e0 2024-02-23T00:31:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:24.620 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 msgr2=0x7f1ea40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 0x7f1ea40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f1eb40096a0 tx=0x7f1eb4006b40).stop 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 msgr2=0x7f1eb80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f1ea8005010 tx=0x7f1ea800b040).stop 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 shutdown_connections 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1ea402ff10 0x7f1ea40323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 --2- 172.21.15.118:0/1206534728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1eb80601e0 0x7f1eb80f69a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 >> 172.21.15.118:0/1206534728 conn(0x7f1eb80f09a0 msgr2=0x7f1eb80f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:24.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 shutdown_connections 2024-02-23T00:31:24.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:24.621+0000 7f1ebe283700 1 -- 172.21.15.118:0/1206534728 wait complete. 2024-02-23T00:31:24.623 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:25.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:24.619855+0000 mon.smithi090 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.118:0/1206534728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:25.187289+0000 mgr.smithi090.pxxmlz (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:25.250137+0000 mon.smithi090 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:31:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:25.257362+0000 mon.smithi090 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:31:26.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:26.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:27.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:27.187598+0000 mgr.smithi090.pxxmlz (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.609+0000 7fe8a6992700 1 -- 172.21.15.118:0/3344799755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 msgr2=0x7fe8a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.609+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3344799755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00f55c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fe890009fb0 tx=0x7fe890009b30).stop 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- 172.21.15.118:0/3344799755 shutdown_connections 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3344799755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00f55c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- 172.21.15.118:0/3344799755 >> 172.21.15.118:0/3344799755 conn(0x7fe8a00f09a0 msgr2=0x7fe8a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- 172.21.15.118:0/3344799755 shutdown_connections 2024-02-23T00:31:28.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- 172.21.15.118:0/3344799755 wait complete. 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 Processor -- start 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- start start 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.610+0000 7fe8a6992700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a00fb6f0 con 0x7fe8a00f51a0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a5990700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a5990700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54708/0 (socket says 172.21.15.118:54708) 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a5990700 1 -- 172.21.15.118:0/3530408671 learned_addr learned my addr 172.21.15.118:0/3530408671 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a5990700 1 -- 172.21.15.118:0/3530408671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890005040 con 0x7fe8a00f51a0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a5990700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe8900089f0 tx=0x7fe89000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe890009320 con 0x7fe8a00f51a0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a00fb890 con 0x7fe8a00f51a0 2024-02-23T00:31:28.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe890013380 con 0x7fe8a00f51a0 2024-02-23T00:31:28.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe890014e20 con 0x7fe8a00f51a0 2024-02-23T00:31:28.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.611+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a00fbd20 con 0x7fe8a00f51a0 2024-02-23T00:31:28.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.612+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0059100 con 0x7fe8a00f51a0 2024-02-23T00:31:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.612+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8900095f0 con 0x7fe8a00f51a0 2024-02-23T00:31:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.612+0000 7fe8967fc700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 0x7fe88c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.612+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe890040e40 con 0x7fe8a00f51a0 2024-02-23T00:31:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.612+0000 7fe8a518f700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 0x7fe88c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.613+0000 7fe8a518f700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 0x7fe88c032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe89c0096a0 tx=0x7fe89c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:28.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.617+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe890014420 con 0x7fe8a00f51a0 2024-02-23T00:31:28.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.824+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8a00fad30 con 0x7fe8a00f51a0 2024-02-23T00:31:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.825+0000 7fe8967fc700 1 -- 172.21.15.118:0/3530408671 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe890010230 con 0x7fe8a00f51a0 2024-02-23T00:31:28.826 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:28.826 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 msgr2=0x7fe88c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 0x7fe88c032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe89c0096a0 tx=0x7fe89c006b40).stop 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 msgr2=0x7fe8a00fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe8900089f0 tx=0x7fe89000b040).stop 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 shutdown_connections 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe88c02ff60 0x7fe88c032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:28.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 --2- 172.21.15.118:0/3530408671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00fd270 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:28.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 >> 172.21.15.118:0/3530408671 conn(0x7fe8a00f09a0 msgr2=0x7fe8a00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:28.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 shutdown_connections 2024-02-23T00:31:28.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:28.827+0000 7fe8a6992700 1 -- 172.21.15.118:0/3530408671 wait complete. 2024-02-23T00:31:28.829 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:29.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:28.825784+0000 mon.smithi090 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.118:0/3530408671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:30.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:29.187976+0000 mgr.smithi090.pxxmlz (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:30.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:30.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:31.188414+0000 mgr.smithi090.pxxmlz (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/865716369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 msgr2=0x7fb5d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/865716369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d00f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fb5c0009fb0 tx=0x7fb5c0009b30).stop 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/865716369 shutdown_connections 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/865716369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d00f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/865716369 >> 172.21.15.118:0/865716369 conn(0x7fb5d00f0b70 msgr2=0x7fb5d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:32.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/865716369 shutdown_connections 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/865716369 wait complete. 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 Processor -- start 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.938+0000 7fb5d4fd7700 1 -- start start 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5d4fd7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5d4fd7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d0105c10 con 0x7fb5d00f5410 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5ce59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:32.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5ce59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:54726/0 (socket says 172.21.15.118:54726) 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5ce59c700 1 -- 172.21.15.118:0/273399111 learned_addr learned my addr 172.21.15.118:0/273399111 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5ce59c700 1 -- 172.21.15.118:0/273399111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0005040 con 0x7fb5d00f5410 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5ce59c700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb5c0005010 tx=0x7fb5c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5c0009320 con 0x7fb5d00f5410 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d0105e10 con 0x7fb5d00f5410 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5c0013380 con 0x7fb5d00f5410 2024-02-23T00:31:32.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5c0014e50 con 0x7fb5d00f5410 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.939+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d01062a0 con 0x7fb5d00f5410 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.940+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5c00095f0 con 0x7fb5d00f5410 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.940+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d0045ad0 con 0x7fb5d00f5410 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.940+0000 7fb5beffd700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 0x7fb5b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.940+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5c0040fa0 con 0x7fb5d00f5410 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.940+0000 7fb5cdd9b700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 0x7fb5b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:32.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.941+0000 7fb5cdd9b700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 0x7fb5b80323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb5c40096a0 tx=0x7fb5c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:32.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:32.945+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5c0014420 con 0x7fb5d00f5410 2024-02-23T00:31:33.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.149+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5d0107150 con 0x7fb5d00f5410 2024-02-23T00:31:33.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.149+0000 7fb5beffd700 1 -- 172.21.15.118:0/273399111 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5c0010230 con 0x7fb5d00f5410 2024-02-23T00:31:33.151 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:33.151 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:33.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 msgr2=0x7fb5b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:33.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 0x7fb5b80323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb5c40096a0 tx=0x7fb5c4006b40).stop 2024-02-23T00:31:33.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 msgr2=0x7fb5d0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:33.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb5c0005010 tx=0x7fb5c000b040).stop 2024-02-23T00:31:33.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 shutdown_connections 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb5b802ff10 0x7fb5b80323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 --2- 172.21.15.118:0/273399111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5d00f5410 0x7fb5d0105720 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 >> 172.21.15.118:0/273399111 conn(0x7fb5d00f0b70 msgr2=0x7fb5d00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 shutdown_connections 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:33.151+0000 7fb5d4fd7700 1 -- 172.21.15.118:0/273399111 wait complete. 2024-02-23T00:31:33.153 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:33.150430+0000 mon.smithi090 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.118:0/273399111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:34.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:33.188804+0000 mgr.smithi090.pxxmlz (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:34.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:34.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:36.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:35.189140+0000 mgr.smithi090.pxxmlz (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- 172.21.15.118:0/2932769335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 msgr2=0x7fae840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 --2- 172.21.15.118:0/2932769335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fae80009fb0 tx=0x7fae80009b30).stop 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- 172.21.15.118:0/2932769335 shutdown_connections 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 --2- 172.21.15.118:0/2932769335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- 172.21.15.118:0/2932769335 >> 172.21.15.118:0/2932769335 conn(0x7fae840f09a0 msgr2=0x7fae840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:37.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- 172.21.15.118:0/2932769335 shutdown_connections 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- 172.21.15.118:0/2932769335 wait complete. 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 Processor -- start 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- start start 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.136+0000 7fae8b02a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae840f13d0 con 0x7fae840f51a0 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8a028700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8a028700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:34132/0 (socket says 172.21.15.118:34132) 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8a028700 1 -- 172.21.15.118:0/4214025428 learned_addr learned my addr 172.21.15.118:0/4214025428 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8a028700 1 -- 172.21.15.118:0/4214025428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80005040 con 0x7fae840f51a0 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8a028700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fae800089f0 tx=0x7fae8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae80009320 con 0x7fae840f51a0 2024-02-23T00:31:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae840f15d0 con 0x7fae840f51a0 2024-02-23T00:31:37.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae80013380 con 0x7fae840f51a0 2024-02-23T00:31:37.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae80014e20 con 0x7fae840f51a0 2024-02-23T00:31:37.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.137+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae840f1a60 con 0x7fae840f51a0 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.138+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae800095f0 con 0x7fae840f51a0 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.138+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae84059100 con 0x7fae840f51a0 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.138+0000 7fae7affd700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.138+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae80040fb0 con 0x7fae840f51a0 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.139+0000 7fae89827700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:37.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.139+0000 7fae89827700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 0x7fae700323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fae740096a0 tx=0x7fae74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:37.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.143+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae8006d2b0 con 0x7fae840f51a0 2024-02-23T00:31:37.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.348+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae84027a00 con 0x7fae840f51a0 2024-02-23T00:31:37.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.349+0000 7fae7affd700 1 -- 172.21.15.118:0/4214025428 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae80014420 con 0x7fae840f51a0 2024-02-23T00:31:37.352 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:37.352 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.350+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 msgr2=0x7fae700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.350+0000 7fae8b02a700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 0x7fae700323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fae740096a0 tx=0x7fae74006b40).stop 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.350+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 msgr2=0x7fae840f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.350+0000 7fae8b02a700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fae800089f0 tx=0x7fae8000b040).stop 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 shutdown_connections 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:37.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 --2- 172.21.15.118:0/4214025428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae840f51a0 0x7fae840f2f50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:37.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 >> 172.21.15.118:0/4214025428 conn(0x7fae840f09a0 msgr2=0x7fae8405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:37.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 shutdown_connections 2024-02-23T00:31:37.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:37.351+0000 7fae8b02a700 1 -- 172.21.15.118:0/4214025428 wait complete. 2024-02-23T00:31:37.354 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:38.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:37.189462+0000 mgr.smithi090.pxxmlz (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:38.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:37.351050+0000 mon.smithi090 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.118:0/4214025428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:39.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:39.062 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:40.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:39.189738+0000 mgr.smithi090.pxxmlz (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:41.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 -- 172.21.15.118:0/3899680810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf80f6510 msgr2=0x7f7cf80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3899680810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf80f6510 0x7f7cf80f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f7ce0009fb0 tx=0x7f7ce0009b30).stop 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 -- 172.21.15.118:0/3899680810 shutdown_connections 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3899680810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf80f6510 0x7f7cf80f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 -- 172.21.15.118:0/3899680810 >> 172.21.15.118:0/3899680810 conn(0x7f7cf80f1c70 msgr2=0x7f7cf80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 -- 172.21.15.118:0/3899680810 shutdown_connections 2024-02-23T00:31:41.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.071+0000 7f7cfd069700 1 -- 172.21.15.118:0/3899680810 wait complete. 2024-02-23T00:31:41.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cfd069700 1 Processor -- start 2024-02-23T00:31:41.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cfd069700 1 -- start start 2024-02-23T00:31:41.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cfd069700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:41.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cfd069700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8109420 con 0x7f7cf8108b10 2024-02-23T00:31:41.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cf6d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cf6d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:34154/0 (socket says 172.21.15.118:34154) 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cf6d9d700 1 -- 172.21.15.118:0/3979277700 learned_addr learned my addr 172.21.15.118:0/3979277700 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.072+0000 7f7cf6d9d700 1 -- 172.21.15.118:0/3979277700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0005040 con 0x7f7cf8108b10 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cf6d9d700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f7cf80f7610 tx=0x7f7ce0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce0013380 con 0x7f7cf8108b10 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8109620 con 0x7f7cf8108b10 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ce0013790 con 0x7f7cf8108b10 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce001ee70 con 0x7f7cf8108b10 2024-02-23T00:31:41.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.073+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf810a170 con 0x7f7cf8108b10 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ce0025060 con 0x7f7cf8108b10 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf8059100 con 0x7f7cf8108b10 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cef7fe700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 0x7f7ce4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ce0040e90 con 0x7f7cf8108b10 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cf659c700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 0x7f7ce4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.074+0000 7f7cf659c700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 0x7f7ce4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7ce80096a0 tx=0x7f7ce8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.079+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ce001c950 con 0x7f7cf8108b10 2024-02-23T00:31:41.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.287+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cf8045ad0 con 0x7f7cf8108b10 2024-02-23T00:31:41.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.288+0000 7f7cef7fe700 1 -- 172.21.15.118:0/3979277700 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ce002d2d0 con 0x7f7cf8108b10 2024-02-23T00:31:41.290 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:41.290 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 msgr2=0x7f7ce4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 0x7f7ce4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7ce80096a0 tx=0x7f7ce8006b40).stop 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 msgr2=0x7f7cf8108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f7cf80f7610 tx=0x7f7ce0009b30).stop 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 shutdown_connections 2024-02-23T00:31:41.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7ce402ff60 0x7f7ce4032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 --2- 172.21.15.118:0/3979277700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cf8108b10 0x7f7cf8108f30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.289+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 >> 172.21.15.118:0/3979277700 conn(0x7f7cf80f1c70 msgr2=0x7f7cf80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.290+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 shutdown_connections 2024-02-23T00:31:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:41.290+0000 7f7cfd069700 1 -- 172.21.15.118:0/3979277700 wait complete. 2024-02-23T00:31:41.293 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:41.190149+0000 mgr.smithi090.pxxmlz (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:41.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:41.289759+0000 mon.smithi090 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.118:0/3979277700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:43.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:43.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:43.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:43.190620+0000 mgr.smithi090.pxxmlz (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:45.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.155+0000 7fefdaed5700 1 -- 172.21.15.118:0/188899898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 msgr2=0x7fefd40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:45.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.155+0000 7fefdaed5700 1 --2- 172.21.15.118:0/188899898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd40f7d00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fefc4009fb0 tx=0x7fefc4009b30).stop 2024-02-23T00:31:45.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.155+0000 7fefdaed5700 1 -- 172.21.15.118:0/188899898 shutdown_connections 2024-02-23T00:31:45.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.155+0000 7fefdaed5700 1 --2- 172.21.15.118:0/188899898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd40f7d00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.155+0000 7fefdaed5700 1 -- 172.21.15.118:0/188899898 >> 172.21.15.118:0/188899898 conn(0x7fefd40f0120 msgr2=0x7fefd40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 -- 172.21.15.118:0/188899898 shutdown_connections 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 -- 172.21.15.118:0/188899898 wait complete. 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 Processor -- start 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 -- start start 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.156+0000 7fefdaed5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd4106af0 con 0x7fefd40f78e0 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefd9ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefd9ed3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:49336/0 (socket says 172.21.15.118:49336) 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefd9ed3700 1 -- 172.21.15.118:0/4250386336 learned_addr learned my addr 172.21.15.118:0/4250386336 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefd9ed3700 1 -- 172.21.15.118:0/4250386336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4005040 con 0x7fefd40f78e0 2024-02-23T00:31:45.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefd9ed3700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fefc40089f0 tx=0x7fefc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefc4009320 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd4106cf0 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefc4013380 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefc4014e20 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.157+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd4107180 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.158+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefc40095f0 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.158+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd4045ad0 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.158+0000 7fefca7fc700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 0x7fefc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.158+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefc4040f50 con 0x7fefd40f78e0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.159+0000 7fefd96d2700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 0x7fefc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:45.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.159+0000 7fefd96d2700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 0x7fefc0032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fefd00096a0 tx=0x7fefd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:45.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.163+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefc4014420 con 0x7fefd40f78e0 2024-02-23T00:31:45.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.372+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefd4108030 con 0x7fefd40f78e0 2024-02-23T00:31:45.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.372+0000 7fefca7fc700 1 -- 172.21.15.118:0/4250386336 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefc401d980 con 0x7fefd40f78e0 2024-02-23T00:31:45.374 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:45.374 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:45.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 msgr2=0x7fefc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:45.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 0x7fefc0032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fefd00096a0 tx=0x7fefd0006b40).stop 2024-02-23T00:31:45.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 msgr2=0x7fefd4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:45.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fefc40089f0 tx=0x7fefc400b040).stop 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 shutdown_connections 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fefc002ff60 0x7fefc0032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 --2- 172.21.15.118:0/4250386336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd40f78e0 0x7fefd4106600 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 >> 172.21.15.118:0/4250386336 conn(0x7fefd40f0120 msgr2=0x7fefd40f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 shutdown_connections 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:45.374+0000 7fefdaed5700 1 -- 172.21.15.118:0/4250386336 wait complete. 2024-02-23T00:31:45.376 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:45.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:45.191069+0000 mgr.smithi090.pxxmlz (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:46.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:45.373303+0000 mon.smithi090 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.118:0/4250386336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:47.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:47.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:47.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:47.191466+0000 mgr.smithi090.pxxmlz (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- 172.21.15.118:0/1569392065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 msgr2=0x7f3d5c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 --2- 172.21.15.118:0/1569392065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c0f55c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f3d4c009fb0 tx=0x7f3d4c009b30).stop 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- 172.21.15.118:0/1569392065 shutdown_connections 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 --2- 172.21.15.118:0/1569392065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c0f55c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- 172.21.15.118:0/1569392065 >> 172.21.15.118:0/1569392065 conn(0x7f3d5c0f09a0 msgr2=0x7f3d5c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:49.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- 172.21.15.118:0/1569392065 shutdown_connections 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- 172.21.15.118:0/1569392065 wait complete. 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 Processor -- start 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- start start 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.288+0000 7f3d6253a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c106ad0 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d61538700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d61538700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:49348/0 (socket says 172.21.15.118:49348) 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d61538700 1 -- 172.21.15.118:0/2890897443 learned_addr learned my addr 172.21.15.118:0/2890897443 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d61538700 1 -- 172.21.15.118:0/2890897443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c005040 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d61538700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f3d4c0089f0 tx=0x7f3d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d4c009320 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c106cd0 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d4c013380 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d4c014e20 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.289+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c107160 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.290+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d4c0095f0 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.290+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c045ad0 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.290+0000 7f3d51ffb700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 0x7f3d480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.290+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d4c040f50 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.291+0000 7f3d60d37700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 0x7f3d480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.291+0000 7f3d60d37700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 0x7f3d480323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:49.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.295+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d4c014420 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.502+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d5c108010 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.503+0000 7f3d51ffb700 1 -- 172.21.15.118:0/2890897443 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d4c01d980 con 0x7f3d5c0f51a0 2024-02-23T00:31:49.505 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:49.505 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:49.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 msgr2=0x7f3d480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 0x7f3d480323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).stop 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 msgr2=0x7f3d5c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f3d4c0089f0 tx=0x7f3d4c00b040).stop 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 shutdown_connections 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3d4802ff10 0x7f3d480323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 --2- 172.21.15.118:0/2890897443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d5c0f51a0 0x7f3d5c1065e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 >> 172.21.15.118:0/2890897443 conn(0x7f3d5c0f09a0 msgr2=0x7f3d5c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 shutdown_connections 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:49.505+0000 7f3d6253a700 1 -- 172.21.15.118:0/2890897443 wait complete. 2024-02-23T00:31:49.507 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:49.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:49.191912+0000 mgr.smithi090.pxxmlz (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:50.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:49.504030+0000 mon.smithi090 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.118:0/2890897443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:51.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:51.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:51.192288+0000 mgr.smithi090.pxxmlz (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.327+0000 7fb89f548700 1 -- 172.21.15.118:0/3216000231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 msgr2=0x7fb8980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.327+0000 7fb89f548700 1 --2- 172.21.15.118:0/3216000231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb8980f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fb888009fb0 tx=0x7fb888009b30).stop 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- 172.21.15.118:0/3216000231 shutdown_connections 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 --2- 172.21.15.118:0/3216000231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb8980f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- 172.21.15.118:0/3216000231 >> 172.21.15.118:0/3216000231 conn(0x7fb8980f1c70 msgr2=0x7fb8980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:53.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- 172.21.15.118:0/3216000231 shutdown_connections 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- 172.21.15.118:0/3216000231 wait complete. 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 Processor -- start 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- start start 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.328+0000 7fb89f548700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898069e20 con 0x7fb8980f6510 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89d2e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89d2e4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:49358/0 (socket says 172.21.15.118:49358) 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89d2e4700 1 -- 172.21.15.118:0/1559778463 learned_addr learned my addr 172.21.15.118:0/1559778463 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89d2e4700 1 -- 172.21.15.118:0/1559778463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888005040 con 0x7fb8980f6510 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89d2e4700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb888005010 tx=0x7fb88800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb888009320 con 0x7fb8980f6510 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb888013380 con 0x7fb8980f6510 2024-02-23T00:31:53.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898069fc0 con 0x7fb8980f6510 2024-02-23T00:31:53.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb888014e50 con 0x7fb8980f6510 2024-02-23T00:31:53.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.329+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89806a450 con 0x7fb8980f6510 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.330+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8880095f0 con 0x7fb8980f6510 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.330+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb898059100 con 0x7fb8980f6510 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.330+0000 7fb88dffb700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 0x7fb8840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.330+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb888040f80 con 0x7fb8980f6510 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.331+0000 7fb89cae3700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 0x7fb8840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:53.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.331+0000 7fb89cae3700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 0x7fb8840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb8940096a0 tx=0x7fb894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:53.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.335+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb888014420 con 0x7fb8980f6510 2024-02-23T00:31:53.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.543+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8980fc0a0 con 0x7fb8980f6510 2024-02-23T00:31:53.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.544+0000 7fb88dffb700 1 -- 172.21.15.118:0/1559778463 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb888010230 con 0x7fb8980f6510 2024-02-23T00:31:53.546 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:53.546 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:53.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 msgr2=0x7fb8840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:53.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 0x7fb8840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb8940096a0 tx=0x7fb894006b40).stop 2024-02-23T00:31:53.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 msgr2=0x7fb89806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:53.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb888005010 tx=0x7fb88800b040).stop 2024-02-23T00:31:53.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 shutdown_connections 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb88402ff10 0x7fb8840323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 --2- 172.21.15.118:0/1559778463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8980f6510 0x7fb89806b9a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 >> 172.21.15.118:0/1559778463 conn(0x7fb8980f1c70 msgr2=0x7fb8980f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 shutdown_connections 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:53.546+0000 7fb89f548700 1 -- 172.21.15.118:0/1559778463 wait complete. 2024-02-23T00:31:53.548 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:53.192667+0000 mgr.smithi090.pxxmlz (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:54.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:53.544844+0000 mon.smithi090 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.118:0/1559778463' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:55.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:55.164 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:55.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:55.192969+0000 mgr.smithi090.pxxmlz (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 -- 172.21.15.118:0/2901026591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 msgr2=0x7fc8700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 --2- 172.21.15.118:0/2901026591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc8700f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fc858009fb0 tx=0x7fc858009b30).stop 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 -- 172.21.15.118:0/2901026591 shutdown_connections 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 --2- 172.21.15.118:0/2901026591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc8700f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 -- 172.21.15.118:0/2901026591 >> 172.21.15.118:0/2901026591 conn(0x7fc8700f0b70 msgr2=0x7fc8700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 -- 172.21.15.118:0/2901026591 shutdown_connections 2024-02-23T00:31:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.253+0000 7fc874fac700 1 -- 172.21.15.118:0/2901026591 wait complete. 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.254+0000 7fc874fac700 1 Processor -- start 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.254+0000 7fc874fac700 1 -- start start 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.254+0000 7fc874fac700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.254+0000 7fc874fac700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc870060220 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc86e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc86e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57720/0 (socket says 172.21.15.118:57720) 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc86e59c700 1 -- 172.21.15.118:0/4223010697 learned_addr learned my addr 172.21.15.118:0/4223010697 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc86e59c700 1 -- 172.21.15.118:0/4223010697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc858005040 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc86e59c700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fc858005010 tx=0x7fc85800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc858009320 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc870060420 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc858013380 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc858014e50 con 0x7fc8700f5410 2024-02-23T00:31:57.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.255+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc870109bb0 con 0x7fc8700f5410 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.256+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8580095f0 con 0x7fc8700f5410 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.256+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc870059100 con 0x7fc8700f5410 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.256+0000 7fc866ffd700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 0x7fc85c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.256+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc858040e90 con 0x7fc8700f5410 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.256+0000 7fc86dd9b700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 0x7fc85c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.257+0000 7fc86dd9b700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 0x7fc85c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc8600096a0 tx=0x7fc860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:57.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.261+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc858014420 con 0x7fc8700f5410 2024-02-23T00:31:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.467+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc870045ad0 con 0x7fc8700f5410 2024-02-23T00:31:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.468+0000 7fc866ffd700 1 -- 172.21.15.118:0/4223010697 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc858010230 con 0x7fc8700f5410 2024-02-23T00:31:57.469 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:31:57.469 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 msgr2=0x7fc85c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 0x7fc85c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc8600096a0 tx=0x7fc860006b40).stop 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 msgr2=0x7fc870109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fc858005010 tx=0x7fc85800b040).stop 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 shutdown_connections 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc85c02ff10 0x7fc85c0323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 --2- 172.21.15.118:0/4223010697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8700f5410 0x7fc870109500 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 >> 172.21.15.118:0/4223010697 conn(0x7fc8700f0b70 msgr2=0x7fc8700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 shutdown_connections 2024-02-23T00:31:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:31:57.469+0000 7fc874fac700 1 -- 172.21.15.118:0/4223010697 wait complete. 2024-02-23T00:31:57.472 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:31:57.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:57.193233+0000 mgr.smithi090.pxxmlz (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:31:58.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:31:57.468573+0000 mon.smithi090 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.118:0/4223010697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:31:59.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:31:59.144 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:31:59.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:31:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:31:59.193518+0000 mgr.smithi090.pxxmlz (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:01.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- 172.21.15.118:0/1062984316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 msgr2=0x7fc03c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:01.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 --2- 172.21.15.118:0/1062984316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c0f66c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc038009fb0 tx=0x7fc038009b30).stop 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- 172.21.15.118:0/1062984316 shutdown_connections 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 --2- 172.21.15.118:0/1062984316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c0f66c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- 172.21.15.118:0/1062984316 >> 172.21.15.118:0/1062984316 conn(0x7fc03c0f1ac0 msgr2=0x7fc03c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- 172.21.15.118:0/1062984316 shutdown_connections 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- 172.21.15.118:0/1062984316 wait complete. 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 Processor -- start 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- start start 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.284+0000 7fc042b58700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c106ab0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc041b56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc041b56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:57724/0 (socket says 172.21.15.118:57724) 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc041b56700 1 -- 172.21.15.118:0/2149572083 learned_addr learned my addr 172.21.15.118:0/2149572083 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:01.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc041b56700 1 -- 172.21.15.118:0/2149572083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc038005040 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc041b56700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc0380089f0 tx=0x7fc03800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc038009320 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c106cb0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc038013380 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc038014e20 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.285+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c107140 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.286+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0380263a0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.286+0000 7fc0327fc700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 0x7fc0280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.286+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc038041170 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.286+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c045ad0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.286+0000 7fc041355700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 0x7fc0280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.287+0000 7fc041355700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 0x7fc0280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc02c0096a0 tx=0x7fc02c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:01.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.291+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc03801ccf0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.496+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc03c107ff0 con 0x7fc03c0f62a0 2024-02-23T00:32:01.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.498+0000 7fc0327fc700 1 -- 172.21.15.118:0/2149572083 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc038012330 con 0x7fc03c0f62a0 2024-02-23T00:32:01.500 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:01.500 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:01.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 msgr2=0x7fc0280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:01.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 0x7fc0280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc02c0096a0 tx=0x7fc02c006b40).stop 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 msgr2=0x7fc03c1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc0380089f0 tx=0x7fc03800b040).stop 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 shutdown_connections 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc02802ff10 0x7fc0280323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 --2- 172.21.15.118:0/2149572083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c0f62a0 0x7fc03c1065c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 >> 172.21.15.118:0/2149572083 conn(0x7fc03c0f1ac0 msgr2=0x7fc03c0f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 shutdown_connections 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:01.500+0000 7fc042b58700 1 -- 172.21.15.118:0/2149572083 wait complete. 2024-02-23T00:32:01.502 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:01.193960+0000 mgr.smithi090.pxxmlz (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:02.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:01.499005+0000 mon.smithi090 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.118:0/2149572083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:03.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:03.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:03.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:03.194368+0000 mgr.smithi090.pxxmlz (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:05.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 -- 172.21.15.118:0/2616119118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 msgr2=0x7f7f600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 --2- 172.21.15.118:0/2616119118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f7f50009fb0 tx=0x7f7f50009b30).stop 2024-02-23T00:32:05.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 -- 172.21.15.118:0/2616119118 shutdown_connections 2024-02-23T00:32:05.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 --2- 172.21.15.118:0/2616119118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 -- 172.21.15.118:0/2616119118 >> 172.21.15.118:0/2616119118 conn(0x7f7f600f0b70 msgr2=0x7f7f600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 -- 172.21.15.118:0/2616119118 shutdown_connections 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.451+0000 7f7f67562700 1 -- 172.21.15.118:0/2616119118 wait complete. 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f67562700 1 Processor -- start 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f67562700 1 -- start start 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f67562700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f67562700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600fb960 con 0x7f7f600f5410 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f652fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f652fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:40050/0 (socket says 172.21.15.118:40050) 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f652fe700 1 -- 172.21.15.118:0/1648706737 learned_addr learned my addr 172.21.15.118:0/1648706737 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.452+0000 7f7f652fe700 1 -- 172.21.15.118:0/1648706737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50005040 con 0x7f7f600f5410 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f652fe700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f7f50005010 tx=0x7f7f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f50009320 con 0x7f7f600f5410 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600fbb00 con 0x7f7f600f5410 2024-02-23T00:32:05.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f50013380 con 0x7f7f600f5410 2024-02-23T00:32:05.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f50014e50 con 0x7f7f600f5410 2024-02-23T00:32:05.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.453+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f600fbf90 con 0x7f7f600f5410 2024-02-23T00:32:05.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.454+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f60059100 con 0x7f7f600f5410 2024-02-23T00:32:05.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.454+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f500095f0 con 0x7f7f600f5410 2024-02-23T00:32:05.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.454+0000 7f7f55ffb700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 0x7f7f4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:05.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.454+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f50040e90 con 0x7f7f600f5410 2024-02-23T00:32:05.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.454+0000 7f7f64afd700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 0x7f7f4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:05.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.455+0000 7f7f64afd700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 0x7f7f4c0323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7f5c0096a0 tx=0x7f7f5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:05.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.459+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f50014420 con 0x7f7f600f5410 2024-02-23T00:32:05.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.664+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f60045ad0 con 0x7f7f600f5410 2024-02-23T00:32:05.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.665+0000 7f7f55ffb700 1 -- 172.21.15.118:0/1648706737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f50010230 con 0x7f7f600f5410 2024-02-23T00:32:05.667 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:05.667 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:05.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 msgr2=0x7f7f4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 0x7f7f4c0323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7f5c0096a0 tx=0x7f7f5c006b40).stop 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 msgr2=0x7f7f600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f7f50005010 tx=0x7f7f5000b040).stop 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 shutdown_connections 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f4c02ff10 0x7f7f4c0323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 --2- 172.21.15.118:0/1648706737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f600f5410 0x7f7f600fd4e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 >> 172.21.15.118:0/1648706737 conn(0x7f7f600f0b70 msgr2=0x7f7f6005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 shutdown_connections 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:05.667+0000 7f7f67562700 1 -- 172.21.15.118:0/1648706737 wait complete. 2024-02-23T00:32:05.669 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:05.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:05.194730+0000 mgr.smithi090.pxxmlz (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:06.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:05.666060+0000 mon.smithi090 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.118:0/1648706737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:07.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:07.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:07.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:07.195128+0000 mgr.smithi090.pxxmlz (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 -- 172.21.15.118:0/1126548569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 msgr2=0x7f385c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 --2- 172.21.15.118:0/1126548569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c0f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3850009fb0 tx=0x7f3850009b30).stop 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 -- 172.21.15.118:0/1126548569 shutdown_connections 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 --2- 172.21.15.118:0/1126548569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c0f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 -- 172.21.15.118:0/1126548569 >> 172.21.15.118:0/1126548569 conn(0x7f385c0f1c70 msgr2=0x7f385c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 -- 172.21.15.118:0/1126548569 shutdown_connections 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.541+0000 7f3862d8c700 1 -- 172.21.15.118:0/1126548569 wait complete. 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.542+0000 7f3862d8c700 1 Processor -- start 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.542+0000 7f3862d8c700 1 -- start start 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.542+0000 7f3862d8c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.542+0000 7f3862d8c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c106cd0 con 0x7f385c0f6510 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.542+0000 7f3860b28700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3860b28700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:40066/0 (socket says 172.21.15.118:40066) 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3860b28700 1 -- 172.21.15.118:0/3537229240 learned_addr learned my addr 172.21.15.118:0/3537229240 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3860b28700 1 -- 172.21.15.118:0/3537229240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850005040 con 0x7f385c0f6510 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3860b28700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3850005010 tx=0x7f385000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3850009320 con 0x7f385c0f6510 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c106ed0 con 0x7f385c0f6510 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3850013380 con 0x7f385c0f6510 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3850014e50 con 0x7f385c0f6510 2024-02-23T00:32:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.543+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c107360 con 0x7f385c0f6510 2024-02-23T00:32:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.544+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3850014420 con 0x7f385c0f6510 2024-02-23T00:32:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.544+0000 7f38597fa700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 0x7f384c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.544+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3850040f40 con 0x7f385c0f6510 2024-02-23T00:32:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.544+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c045ad0 con 0x7f385c0f6510 2024-02-23T00:32:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.544+0000 7f385bfff700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 0x7f384c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.545+0000 7f385bfff700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 0x7f384c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.549+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38500095f0 con 0x7f385c0f6510 2024-02-23T00:32:09.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.755+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f385c108210 con 0x7f385c0f6510 2024-02-23T00:32:09.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.756+0000 7f38597fa700 1 -- 172.21.15.118:0/3537229240 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f385002b020 con 0x7f385c0f6510 2024-02-23T00:32:09.757 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:09.757 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 msgr2=0x7f384c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 0x7f384c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).stop 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 msgr2=0x7f385c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3850005010 tx=0x7f385000b040).stop 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 shutdown_connections 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f384c02ff10 0x7f384c0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 --2- 172.21.15.118:0/3537229240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f385c0f6510 0x7f385c1067e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 >> 172.21.15.118:0/3537229240 conn(0x7f385c0f1c70 msgr2=0x7f385c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 shutdown_connections 2024-02-23T00:32:09.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:09.757+0000 7f3862d8c700 1 -- 172.21.15.118:0/3537229240 wait complete. 2024-02-23T00:32:09.760 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:09.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:09.195425+0000 mgr.smithi090.pxxmlz (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:10.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:09.756588+0000 mon.smithi090 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.118:0/3537229240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:11.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:11.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:11.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:11.195757+0000 mgr.smithi090.pxxmlz (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:13.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 -- 172.21.15.118:0/1834256208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 msgr2=0x7f68fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:13.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 --2- 172.21.15.118:0/1834256208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc0f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f68ec009fb0 tx=0x7f68ec009b30).stop 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 -- 172.21.15.118:0/1834256208 shutdown_connections 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 --2- 172.21.15.118:0/1834256208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc0f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 -- 172.21.15.118:0/1834256208 >> 172.21.15.118:0/1834256208 conn(0x7f68fc0f1c70 msgr2=0x7f68fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 -- 172.21.15.118:0/1834256208 shutdown_connections 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.634+0000 7f6901d2b700 1 -- 172.21.15.118:0/1834256208 wait complete. 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f6901d2b700 1 Processor -- start 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f6901d2b700 1 -- start start 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f6901d2b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f6901d2b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc069e20 con 0x7f68fc0f6510 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f68fb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f68fb7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:40094/0 (socket says 172.21.15.118:40094) 2024-02-23T00:32:13.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.635+0000 7f68fb7fe700 1 -- 172.21.15.118:0/3386037924 learned_addr learned my addr 172.21.15.118:0/3386037924 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f68fb7fe700 1 -- 172.21.15.118:0/3386037924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec005040 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f68fb7fe700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f68ec005010 tx=0x7f68ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68ec009320 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc069fc0 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68ec013380 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68ec014e50 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.636+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68fc06a450 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc059100 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68ec014420 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f68e3fff700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 0x7f68e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68ec040e60 con 0x7f68fc0f6510 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f68faffd700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 0x7f68e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:13.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.637+0000 7f68faffd700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 0x7f68e40323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f68f00096a0 tx=0x7f68f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:13.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.641+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68ec0095f0 con 0x7f68fc0f6510 2024-02-23T00:32:13.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:13.196161+0000 mgr.smithi090.pxxmlz (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:13.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.847+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68fc045ad0 con 0x7f68fc0f6510 2024-02-23T00:32:13.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.847+0000 7f68e3fff700 1 -- 172.21.15.118:0/3386037924 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68ec01db70 con 0x7f68fc0f6510 2024-02-23T00:32:13.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:13.849 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:13.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 msgr2=0x7f68e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:13.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 0x7f68e40323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f68f00096a0 tx=0x7f68f0006b40).stop 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 msgr2=0x7f68fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f68ec005010 tx=0x7f68ec00b040).stop 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 shutdown_connections 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f68e402ff10 0x7f68e40323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 --2- 172.21.15.118:0/3386037924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68fc0f6510 0x7f68fc06b9a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 >> 172.21.15.118:0/3386037924 conn(0x7f68fc0f1c70 msgr2=0x7f68fc0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:13.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 shutdown_connections 2024-02-23T00:32:13.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:13.849+0000 7f6901d2b700 1 -- 172.21.15.118:0/3386037924 wait complete. 2024-02-23T00:32:13.853 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:14.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:13.848326+0000 mon.smithi090 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.118:0/3386037924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:15.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:15.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:15.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:15.196561+0000 mgr.smithi090.pxxmlz (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:17.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:17.196850+0000 mgr.smithi090.pxxmlz (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:17.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 -- 172.21.15.118:0/1111935748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 msgr2=0x7fe6280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:17.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 --2- 172.21.15.118:0/1111935748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6280f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fe61c009fb0 tx=0x7fe61c009b30).stop 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 -- 172.21.15.118:0/1111935748 shutdown_connections 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 --2- 172.21.15.118:0/1111935748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6280f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 -- 172.21.15.118:0/1111935748 >> 172.21.15.118:0/1111935748 conn(0x7fe6280f0b70 msgr2=0x7fe6280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 -- 172.21.15.118:0/1111935748 shutdown_connections 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.870+0000 7fe62feb0700 1 -- 172.21.15.118:0/1111935748 wait complete. 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62feb0700 1 Processor -- start 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62feb0700 1 -- start start 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62feb0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62feb0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe628106ca0 con 0x7fe6280f5410 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62dc4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62dc4c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:52358/0 (socket says 172.21.15.118:52358) 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62dc4c700 1 -- 172.21.15.118:0/3423267917 learned_addr learned my addr 172.21.15.118:0/3423267917 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:17.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.871+0000 7fe62dc4c700 1 -- 172.21.15.118:0/3423267917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c005040 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe62dc4c700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe61c005010 tx=0x7fe61c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c009320 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe628106ea0 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe61c013380 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61c014e50 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.872+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe628107330 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe61c014420 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe628045ad0 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe61a7fc700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe61c040e50 con 0x7fe6280f5410 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe62d44b700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.873+0000 7fe62d44b700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 0x7fe6140323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe6240096a0 tx=0x7fe624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:17.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:17.877+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe61c0095f0 con 0x7fe6280f5410 2024-02-23T00:32:18.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.085+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6281081e0 con 0x7fe6280f5410 2024-02-23T00:32:18.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.086+0000 7fe61a7fc700 1 -- 172.21.15.118:0/3423267917 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe61c01db70 con 0x7fe6280f5410 2024-02-23T00:32:18.087 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:18.088 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:18.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 msgr2=0x7fe6140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:18.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 0x7fe6140323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe6240096a0 tx=0x7fe624006b40).stop 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 msgr2=0x7fe6281067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe61c005010 tx=0x7fe61c00b040).stop 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 shutdown_connections 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 --2- 172.21.15.118:0/3423267917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6280f5410 0x7fe6281067b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 >> 172.21.15.118:0/3423267917 conn(0x7fe6280f0b70 msgr2=0x7fe62805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 shutdown_connections 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:18.088+0000 7fe62feb0700 1 -- 172.21.15.118:0/3423267917 wait complete. 2024-02-23T00:32:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:18.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:18.087024+0000 mon.smithi090 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.118:0/3423267917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:19.567 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:19.197194+0000 mgr.smithi090.pxxmlz (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:19.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:19.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:21.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:21.197573+0000 mgr.smithi090.pxxmlz (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:21.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 -- 172.21.15.118:0/731638217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480f5410 msgr2=0x7f7f480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/731638217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480f5410 0x7f7f480f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f7f38009fb0 tx=0x7f7f38009b30).stop 2024-02-23T00:32:21.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 -- 172.21.15.118:0/731638217 shutdown_connections 2024-02-23T00:32:21.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/731638217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480f5410 0x7f7f480f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 -- 172.21.15.118:0/731638217 >> 172.21.15.118:0/731638217 conn(0x7f7f480f0b70 msgr2=0x7f7f480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:21.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 -- 172.21.15.118:0/731638217 shutdown_connections 2024-02-23T00:32:21.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.801+0000 7f7f4f51e700 1 -- 172.21.15.118:0/731638217 wait complete. 2024-02-23T00:32:21.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4f51e700 1 Processor -- start 2024-02-23T00:32:21.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4f51e700 1 -- start start 2024-02-23T00:32:21.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4f51e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4f51e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480fd940 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4d2ba700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4d2ba700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:52368/0 (socket says 172.21.15.118:52368) 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4d2ba700 1 -- 172.21.15.118:0/853082268 learned_addr learned my addr 172.21.15.118:0/853082268 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.802+0000 7f7f4d2ba700 1 -- 172.21.15.118:0/853082268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38005040 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f4d2ba700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f7f480f6510 tx=0x7f7f38009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f38013380 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f480fbed0 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f38013790 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f3801ee70 con 0x7f7f480fd520 2024-02-23T00:32:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.803+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f480fc3c0 con 0x7f7f480fd520 2024-02-23T00:32:21.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.804+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f38025060 con 0x7f7f480fd520 2024-02-23T00:32:21.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.804+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f48045ad0 con 0x7f7f480fd520 2024-02-23T00:32:21.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.804+0000 7f7f3dffb700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 0x7f7f34032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.804+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f38040d40 con 0x7f7f480fd520 2024-02-23T00:32:21.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.804+0000 7f7f4cab9700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 0x7f7f34032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.805+0000 7f7f4cab9700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 0x7f7f34032470 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:21.809+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f3801c520 con 0x7f7f480fd520 2024-02-23T00:32:22.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.012+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f4810cb00 con 0x7f7f480fd520 2024-02-23T00:32:22.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.013+0000 7f7f3dffb700 1 -- 172.21.15.118:0/853082268 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f3802d2d0 con 0x7f7f480fd520 2024-02-23T00:32:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:22.016 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 msgr2=0x7f7f34032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 0x7f7f34032470 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).stop 2024-02-23T00:32:22.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 msgr2=0x7f7f480fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f7f480f6510 tx=0x7f7f38009b30).stop 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 shutdown_connections 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7f3402ffb0 0x7f7f34032470 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 --2- 172.21.15.118:0/853082268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f480fd520 0x7f7f480fb9e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 >> 172.21.15.118:0/853082268 conn(0x7f7f480f0b70 msgr2=0x7f7f4805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:22.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 shutdown_connections 2024-02-23T00:32:22.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:22.015+0000 7f7f4f51e700 1 -- 172.21.15.118:0/853082268 wait complete. 2024-02-23T00:32:22.020 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:22.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:22.015026+0000 mon.smithi090 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.118:0/853082268' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:23.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:23.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:23.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:23.197949+0000 mgr.smithi090.pxxmlz (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:25.198252+0000 mgr.smithi090.pxxmlz (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:25.251782+0000 mon.smithi090 (mon.0) 266 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:32:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:25.258261+0000 mon.smithi090 (mon.0) 267 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:32:25.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/781724932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 msgr2=0x7fb8e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/781724932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e40f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fb8d4009fb0 tx=0x7fb8d4009b30).stop 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/781724932 shutdown_connections 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/781724932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e40f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/781724932 >> 172.21.15.118:0/781724932 conn(0x7fb8e40f1c70 msgr2=0x7fb8e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/781724932 shutdown_connections 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/781724932 wait complete. 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.890+0000 7fb8ebfa8700 1 Processor -- start 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8ebfa8700 1 -- start start 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8ebfa8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8ebfa8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e4069e20 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8e9d44700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8e9d44700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42680/0 (socket says 172.21.15.118:42680) 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8e9d44700 1 -- 172.21.15.118:0/3913817268 learned_addr learned my addr 172.21.15.118:0/3913817268 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8e9d44700 1 -- 172.21.15.118:0/3913817268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4005040 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8e9d44700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8d40089f0 tx=0x7fb8d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8d4009320 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e4069fc0 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8d4013380 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8d401ed80 con 0x7fb8e40f6510 2024-02-23T00:32:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.891+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e406a450 con 0x7fb8e40f6510 2024-02-23T00:32:25.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.892+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8d401e420 con 0x7fb8e40f6510 2024-02-23T00:32:25.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.892+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e4059100 con 0x7fb8e40f6510 2024-02-23T00:32:25.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.892+0000 7fb8da7fc700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 0x7fb8d0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:25.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.892+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8d4041000 con 0x7fb8e40f6510 2024-02-23T00:32:25.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.893+0000 7fb8e9543700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 0x7fb8d0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.893+0000 7fb8e9543700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 0x7fb8d0032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb8e00096a0 tx=0x7fb8e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:25.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:25.898+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8d40095f0 con 0x7fb8e40f6510 2024-02-23T00:32:26.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.102+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8e40fc0a0 con 0x7fb8e40f6510 2024-02-23T00:32:26.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8da7fc700 1 -- 172.21.15.118:0/3913817268 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8d401cb10 con 0x7fb8e40f6510 2024-02-23T00:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 msgr2=0x7fb8d0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 0x7fb8d0032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb8e00096a0 tx=0x7fb8e0006b40).stop 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 msgr2=0x7fb8e406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8d40089f0 tx=0x7fb8d400b040).stop 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 shutdown_connections 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb8d002ffb0 0x7fb8d0032470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 --2- 172.21.15.118:0/3913817268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e40f6510 0x7fb8e406b9a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 >> 172.21.15.118:0/3913817268 conn(0x7fb8e40f1c70 msgr2=0x7fb8e40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 shutdown_connections 2024-02-23T00:32:26.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:26.104+0000 7fb8ebfa8700 1 -- 172.21.15.118:0/3913817268 wait complete. 2024-02-23T00:32:26.108 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:26.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:26.104611+0000 mon.smithi090 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.118:0/3913817268' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:27.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:27.198600+0000 mgr.smithi090.pxxmlz (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:27.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:27.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:29.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:29.198917+0000 mgr.smithi090.pxxmlz (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 -- 172.21.15.118:0/3049323517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 msgr2=0x7f40200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 --2- 172.21.15.118:0/3049323517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40200f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4014009fb0 tx=0x7f4014009b30).stop 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 -- 172.21.15.118:0/3049323517 shutdown_connections 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 --2- 172.21.15.118:0/3049323517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40200f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 -- 172.21.15.118:0/3049323517 >> 172.21.15.118:0/3049323517 conn(0x7f40200f0b70 msgr2=0x7f40200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 -- 172.21.15.118:0/3049323517 shutdown_connections 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.960+0000 7f4025fd1700 1 -- 172.21.15.118:0/3049323517 wait complete. 2024-02-23T00:32:29.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f4025fd1700 1 Processor -- start 2024-02-23T00:32:29.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f4025fd1700 1 -- start start 2024-02-23T00:32:29.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f4025fd1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f4025fd1700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4020105bd0 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f401f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f401f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42684/0 (socket says 172.21.15.118:42684) 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.961+0000 7f401f7fe700 1 -- 172.21.15.118:0/2112315732 learned_addr learned my addr 172.21.15.118:0/2112315732 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f401f7fe700 1 -- 172.21.15.118:0/2112315732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4014005040 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f401f7fe700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4014005010 tx=0x7f401400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4014009320 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4020105dd0 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4014013380 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4014014e50 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.962+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4020106260 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.963+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4014014420 con 0x7f40200f5410 2024-02-23T00:32:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.963+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020045ad0 con 0x7f40200f5410 2024-02-23T00:32:29.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.963+0000 7f4007fff700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 0x7f40080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.963+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4014040f70 con 0x7f40200f5410 2024-02-23T00:32:29.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.963+0000 7f401effd700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 0x7f40080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.964+0000 7f401effd700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 0x7f40080323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40100096a0 tx=0x7f4010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:29.968+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40140095f0 con 0x7f40200f5410 2024-02-23T00:32:30.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.177+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40201070b0 con 0x7f40200f5410 2024-02-23T00:32:30.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.178+0000 7f4007fff700 1 -- 172.21.15.118:0/2112315732 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f401401db70 con 0x7f40200f5410 2024-02-23T00:32:30.179 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:30.179 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 msgr2=0x7f40080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 0x7f40080323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f40100096a0 tx=0x7f4010006b40).stop 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 msgr2=0x7f40201056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4014005010 tx=0x7f401400b040).stop 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 shutdown_connections 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f400802ff10 0x7f40080323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 --2- 172.21.15.118:0/2112315732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40200f5410 0x7f40201056e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 >> 172.21.15.118:0/2112315732 conn(0x7f40200f0b70 msgr2=0x7f40200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 shutdown_connections 2024-02-23T00:32:30.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:30.179+0000 7f4025fd1700 1 -- 172.21.15.118:0/2112315732 wait complete. 2024-02-23T00:32:30.182 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:30.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:30.178434+0000 mon.smithi090 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.118:0/2112315732' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:31.199278+0000 mgr.smithi090.pxxmlz (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:31.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:32.000 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:33.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:33.199628+0000 mgr.smithi090.pxxmlz (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:34.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.023+0000 7fe7476fc700 1 -- 172.21.15.118:0/2326121175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 msgr2=0x7fe7400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.023+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2326121175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7400f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fe734009fb0 tx=0x7fe734009b30).stop 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.023+0000 7fe7476fc700 1 -- 172.21.15.118:0/2326121175 shutdown_connections 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.023+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2326121175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7400f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.023+0000 7fe7476fc700 1 -- 172.21.15.118:0/2326121175 >> 172.21.15.118:0/2326121175 conn(0x7fe7400f0b70 msgr2=0x7fe7400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 -- 172.21.15.118:0/2326121175 shutdown_connections 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 -- 172.21.15.118:0/2326121175 wait complete. 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 Processor -- start 2024-02-23T00:32:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 -- start start 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe7476fc700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740060240 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.024+0000 7fe745498700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe745498700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42704/0 (socket says 172.21.15.118:42704) 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe745498700 1 -- 172.21.15.118:0/2068083995 learned_addr learned my addr 172.21.15.118:0/2068083995 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe745498700 1 -- 172.21.15.118:0/2068083995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734005040 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe745498700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe734005010 tx=0x7fe73400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe734009320 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe734013380 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe740060440 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe734014e50 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.025+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe740109b90 con 0x7fe7400f5410 2024-02-23T00:32:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.026+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7340095f0 con 0x7fe7400f5410 2024-02-23T00:32:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.026+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe740045ad0 con 0x7fe7400f5410 2024-02-23T00:32:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.026+0000 7fe731ffb700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.026+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe734040fa0 con 0x7fe7400f5410 2024-02-23T00:32:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.027+0000 7fe744c97700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.027+0000 7fe744c97700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 0x7fe72c0323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe73c0096a0 tx=0x7fe73c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:34.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.031+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe734014420 con 0x7fe7400f5410 2024-02-23T00:32:34.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.237+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe74010a3b0 con 0x7fe7400f5410 2024-02-23T00:32:34.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.238+0000 7fe731ffb700 1 -- 172.21.15.118:0/2068083995 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe734010230 con 0x7fe7400f5410 2024-02-23T00:32:34.239 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:34.240 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:34.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 msgr2=0x7fe72c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 0x7fe72c0323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fe73c0096a0 tx=0x7fe73c006b40).stop 2024-02-23T00:32:34.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 msgr2=0x7fe7401094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe734005010 tx=0x7fe73400b040).stop 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 shutdown_connections 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe72c02ff10 0x7fe72c0323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 --2- 172.21.15.118:0/2068083995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7400f5410 0x7fe7401094e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 >> 172.21.15.118:0/2068083995 conn(0x7fe7400f0b70 msgr2=0x7fe7400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:34.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 shutdown_connections 2024-02-23T00:32:34.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:34.239+0000 7fe7476fc700 1 -- 172.21.15.118:0/2068083995 wait complete. 2024-02-23T00:32:34.244 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:34.777 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:34.238538+0000 mon.smithi090 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.118:0/2068083995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:35.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:35.200078+0000 mgr.smithi090.pxxmlz (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:36.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:36.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:37.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:37.200494+0000 mgr.smithi090.pxxmlz (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:38.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 -- 172.21.15.118:0/3982179281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 msgr2=0x7f59b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:38.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 --2- 172.21.15.118:0/3982179281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b00f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f59a0009fb0 tx=0x7f59a0009b30).stop 2024-02-23T00:32:38.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 -- 172.21.15.118:0/3982179281 shutdown_connections 2024-02-23T00:32:38.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 --2- 172.21.15.118:0/3982179281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b00f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 -- 172.21.15.118:0/3982179281 >> 172.21.15.118:0/3982179281 conn(0x7f59b00f1c70 msgr2=0x7f59b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.511+0000 7f59b77c7700 1 -- 172.21.15.118:0/3982179281 shutdown_connections 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b77c7700 1 -- 172.21.15.118:0/3982179281 wait complete. 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b77c7700 1 Processor -- start 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b77c7700 1 -- start start 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b77c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b77c7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0106cd0 con 0x7f59b00f6510 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b5563700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b5563700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39140/0 (socket says 172.21.15.118:39140) 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.512+0000 7f59b5563700 1 -- 172.21.15.118:0/1269435343 learned_addr learned my addr 172.21.15.118:0/1269435343 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59b5563700 1 -- 172.21.15.118:0/1269435343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a0005040 con 0x7f59b00f6510 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59b5563700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f59a0005010 tx=0x7f59a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a0009320 con 0x7f59b00f6510 2024-02-23T00:32:38.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b0106ed0 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59a0013380 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a0014e50 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.513+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b0107360 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.514+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59a00263a0 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.514+0000 7f59a5ffb700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 0x7f599c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.514+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59a0040f70 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.514+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b0045ad0 con 0x7f59b00f6510 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.514+0000 7f59b4d62700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 0x7f599c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.515+0000 7f59b4d62700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 0x7f599c0323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f59ac0096a0 tx=0x7f59ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:38.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.519+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59a0012920 con 0x7f59b00f6510 2024-02-23T00:32:38.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.728+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59b01080a0 con 0x7f59b00f6510 2024-02-23T00:32:38.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.728+0000 7f59a5ffb700 1 -- 172.21.15.118:0/1269435343 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59a002b020 con 0x7f59b00f6510 2024-02-23T00:32:38.730 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:38.730 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:38.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 msgr2=0x7f599c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:38.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 0x7f599c0323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f59ac0096a0 tx=0x7f59ac006b40).stop 2024-02-23T00:32:38.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 msgr2=0x7f59b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:38.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f59a0005010 tx=0x7f59a000b040).stop 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 shutdown_connections 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f599c02ff10 0x7f599c0323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 --2- 172.21.15.118:0/1269435343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b00f6510 0x7f59b01067e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 >> 172.21.15.118:0/1269435343 conn(0x7f59b00f1c70 msgr2=0x7f59b00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 shutdown_connections 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:38.730+0000 7f59b77c7700 1 -- 172.21.15.118:0/1269435343 wait complete. 2024-02-23T00:32:38.732 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:39.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:38.729239+0000 mon.smithi090 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.118:0/1269435343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:40.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:39.200861+0000 mgr.smithi090.pxxmlz (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:40.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:40.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:41.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:41.201173+0000 mgr.smithi090.pxxmlz (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:42.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 -- 172.21.15.118:0/2481901085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 msgr2=0x7fb1240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:42.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 --2- 172.21.15.118:0/2481901085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1240f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fb11c009960 tx=0x7fb11c004d10).stop 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 -- 172.21.15.118:0/2481901085 shutdown_connections 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 --2- 172.21.15.118:0/2481901085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1240f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 -- 172.21.15.118:0/2481901085 >> 172.21.15.118:0/2481901085 conn(0x7fb1240f1c70 msgr2=0x7fb1240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 -- 172.21.15.118:0/2481901085 shutdown_connections 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.708+0000 7fb12b128700 1 -- 172.21.15.118:0/2481901085 wait complete. 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.709+0000 7fb12b128700 1 Processor -- start 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.709+0000 7fb12b128700 1 -- start start 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.709+0000 7fb12b128700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:42.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.709+0000 7fb12b128700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb124106cd0 con 0x7fb1240f6510 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb128ec4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb128ec4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39150/0 (socket says 172.21.15.118:39150) 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb128ec4700 1 -- 172.21.15.118:0/3695414735 learned_addr learned my addr 172.21.15.118:0/3695414735 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb128ec4700 1 -- 172.21.15.118:0/3695414735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c009640 con 0x7fb1240f6510 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb128ec4700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb11c004b70 tx=0x7fb11c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:42.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb11c018070 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb11c00b970 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb124106ed0 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb11c01c560 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.710+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb124107360 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.711+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb11c013630 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.711+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124045ad0 con 0x7fb1240f6510 2024-02-23T00:32:42.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.711+0000 7fb1217fa700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 0x7fb10c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:42.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.711+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb11c041320 con 0x7fb1240f6510 2024-02-23T00:32:42.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.711+0000 7fb123fff700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 0x7fb10c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:42.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.712+0000 7fb123fff700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 0x7fb10c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb1180096a0 tx=0x7fb118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:42.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.716+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb11c012970 con 0x7fb1240f6510 2024-02-23T00:32:42.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.922+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb124108210 con 0x7fb1240f6510 2024-02-23T00:32:42.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.922+0000 7fb1217fa700 1 -- 172.21.15.118:0/3695414735 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb11c012380 con 0x7fb1240f6510 2024-02-23T00:32:42.924 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:42.924 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:42.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 msgr2=0x7fb10c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 0x7fb10c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb1180096a0 tx=0x7fb118006b40).stop 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 msgr2=0x7fb1241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb11c004b70 tx=0x7fb11c004d10).stop 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 shutdown_connections 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fb10c02ff60 0x7fb10c032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 --2- 172.21.15.118:0/3695414735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1240f6510 0x7fb1241067e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 >> 172.21.15.118:0/3695414735 conn(0x7fb1240f1c70 msgr2=0x7fb1240f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 shutdown_connections 2024-02-23T00:32:42.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:42.924+0000 7fb12b128700 1 -- 172.21.15.118:0/3695414735 wait complete. 2024-02-23T00:32:42.927 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:43.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:42.923315+0000 mon.smithi090 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.118:0/3695414735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:44.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:43.201488+0000 mgr.smithi090.pxxmlz (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:44.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:44.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:45.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:45.201875+0000 mgr.smithi090.pxxmlz (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.989+0000 7f78cccf6700 1 -- 172.21.15.118:0/2783043801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 msgr2=0x7f78c80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.989+0000 7f78cccf6700 1 --2- 172.21.15.118:0/2783043801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f4be0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f78b8009fb0 tx=0x7f78b8009b30).stop 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.989+0000 7f78cccf6700 1 -- 172.21.15.118:0/2783043801 shutdown_connections 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.989+0000 7f78cccf6700 1 --2- 172.21.15.118:0/2783043801 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f4be0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.989+0000 7f78cccf6700 1 -- 172.21.15.118:0/2783043801 >> 172.21.15.118:0/2783043801 conn(0x7f78c80f02e0 msgr2=0x7f78c80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 -- 172.21.15.118:0/2783043801 shutdown_connections 2024-02-23T00:32:46.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 -- 172.21.15.118:0/2783043801 wait complete. 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 Processor -- start 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 -- start start 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78cccf6700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80f57f0 con 0x7f78c80f47c0 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78c659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48950/0 (socket says 172.21.15.118:48950) 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.990+0000 7f78c659c700 1 -- 172.21.15.118:0/4064440004 learned_addr learned my addr 172.21.15.118:0/4064440004 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:46.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78c659c700 1 -- 172.21.15.118:0/4064440004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8005040 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78c659c700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f78b8005010 tx=0x7f78b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b8009320 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c80f5990 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78b8013380 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78b8014e50 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.991+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c80f5e20 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.992+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78b80095f0 con 0x7f78c80f47c0 2024-02-23T00:32:46.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.992+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c8059100 con 0x7f78c80f47c0 2024-02-23T00:32:46.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.992+0000 7f78beffd700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 0x7f78b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:46.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.992+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78b8040e90 con 0x7f78c80f47c0 2024-02-23T00:32:46.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.992+0000 7f78c5d9b700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 0x7f78b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:46.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.993+0000 7f78c5d9b700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 0x7f78b40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f78b00096a0 tx=0x7f78b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:46.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:46.997+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78b8014420 con 0x7f78c80f47c0 2024-02-23T00:32:47.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.202+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78c80fc090 con 0x7f78c80f47c0 2024-02-23T00:32:47.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.203+0000 7f78beffd700 1 -- 172.21.15.118:0/4064440004 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f78b8010230 con 0x7f78c80f47c0 2024-02-23T00:32:47.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:47.204 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 msgr2=0x7f78b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 0x7f78b40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f78b00096a0 tx=0x7f78b0006b40).stop 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 msgr2=0x7f78c80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f78b8005010 tx=0x7f78b800b040).stop 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 shutdown_connections 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f78b402ff10 0x7f78b40323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 --2- 172.21.15.118:0/4064440004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c80f47c0 0x7f78c80f7370 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 >> 172.21.15.118:0/4064440004 conn(0x7f78c80f02e0 msgr2=0x7f78c80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 shutdown_connections 2024-02-23T00:32:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:47.204+0000 7f78cccf6700 1 -- 172.21.15.118:0/4064440004 wait complete. 2024-02-23T00:32:47.207 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:47.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:47.202258+0000 mgr.smithi090.pxxmlz (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:47.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:47.203823+0000 mon.smithi090 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.118:0/4064440004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:49.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:49.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:49.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:49.202561+0000 mgr.smithi090.pxxmlz (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- 172.21.15.118:0/1277522385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 msgr2=0x7fc1b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/1277522385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b00f66c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fc1a0009fb0 tx=0x7fc1a0009b30).stop 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- 172.21.15.118:0/1277522385 shutdown_connections 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/1277522385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b00f66c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- 172.21.15.118:0/1277522385 >> 172.21.15.118:0/1277522385 conn(0x7fc1b00f1aa0 msgr2=0x7fc1b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- 172.21.15.118:0/1277522385 shutdown_connections 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- 172.21.15.118:0/1277522385 wait complete. 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 Processor -- start 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.044+0000 7fc1b62c3700 1 -- start start 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b62c3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b62c3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0106ac0 con 0x7fc1b00f62a0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b52c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b52c1700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48974/0 (socket says 172.21.15.118:48974) 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b52c1700 1 -- 172.21.15.118:0/3904300605 learned_addr learned my addr 172.21.15.118:0/3904300605 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b52c1700 1 -- 172.21.15.118:0/3904300605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a0005040 con 0x7fc1b00f62a0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b52c1700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc1a00089f0 tx=0x7fc1a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a0009320 con 0x7fc1b00f62a0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b0106cc0 con 0x7fc1b00f62a0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1a0013380 con 0x7fc1b00f62a0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.045+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1a0014e20 con 0x7fc1b00f62a0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.046+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b0107150 con 0x7fc1b00f62a0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.046+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1a00263a0 con 0x7fc1b00f62a0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.047+0000 7fc1a5ffb700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 0x7fc19c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.047+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b0045ad0 con 0x7fc1b00f62a0 2024-02-23T00:32:51.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.047+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1a0040f90 con 0x7fc1b00f62a0 2024-02-23T00:32:51.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.047+0000 7fc1b4ac0700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 0x7fc19c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.047+0000 7fc1b4ac0700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 0x7fc19c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc1ac0096a0 tx=0x7fc1ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.052+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1a0012920 con 0x7fc1b00f62a0 2024-02-23T00:32:51.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.255+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1b0107fa0 con 0x7fc1b00f62a0 2024-02-23T00:32:51.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.256+0000 7fc1a5ffb700 1 -- 172.21.15.118:0/3904300605 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1a002b020 con 0x7fc1b00f62a0 2024-02-23T00:32:51.258 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:51.258 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 msgr2=0x7fc19c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 0x7fc19c0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc1ac0096a0 tx=0x7fc1ac006b40).stop 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 msgr2=0x7fc1b01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc1a00089f0 tx=0x7fc1a000b040).stop 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 shutdown_connections 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc19c02ff10 0x7fc19c0323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 --2- 172.21.15.118:0/3904300605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1b00f62a0 0x7fc1b01065d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.257+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 >> 172.21.15.118:0/3904300605 conn(0x7fc1b00f1aa0 msgr2=0x7fc1b00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:51.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.258+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 shutdown_connections 2024-02-23T00:32:51.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:51.258+0000 7fc1b62c3700 1 -- 172.21.15.118:0/3904300605 wait complete. 2024-02-23T00:32:51.260 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:51.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:51.202890+0000 mgr.smithi090.pxxmlz (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:51.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:51.256774+0000 mon.smithi090 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.118:0/3904300605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:53.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:53.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:53.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:53.203241+0000 mgr.smithi090.pxxmlz (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:55.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 -- 172.21.15.118:0/628585079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 msgr2=0x7fc8400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 --2- 172.21.15.118:0/628585079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8400f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fc830009fb0 tx=0x7fc830009b30).stop 2024-02-23T00:32:55.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 -- 172.21.15.118:0/628585079 shutdown_connections 2024-02-23T00:32:55.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 --2- 172.21.15.118:0/628585079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8400f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 -- 172.21.15.118:0/628585079 >> 172.21.15.118:0/628585079 conn(0x7fc8400f0b70 msgr2=0x7fc8400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:55.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 -- 172.21.15.118:0/628585079 shutdown_connections 2024-02-23T00:32:55.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.186+0000 7fc845448700 1 -- 172.21.15.118:0/628585079 wait complete. 2024-02-23T00:32:55.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.187+0000 7fc845448700 1 Processor -- start 2024-02-23T00:32:55.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.187+0000 7fc845448700 1 -- start start 2024-02-23T00:32:55.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.187+0000 7fc845448700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.187+0000 7fc845448700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc840106c90 con 0x7fc840060280 2024-02-23T00:32:55.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc83effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc83effd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:37616/0 (socket says 172.21.15.118:37616) 2024-02-23T00:32:55.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc83effd700 1 -- 172.21.15.118:0/1360591471 learned_addr learned my addr 172.21.15.118:0/1360591471 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:55.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc83effd700 1 -- 172.21.15.118:0/1360591471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc830005040 con 0x7fc840060280 2024-02-23T00:32:55.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc83effd700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc8300089f0 tx=0x7fc83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc830009320 con 0x7fc840060280 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc840106e90 con 0x7fc840060280 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc830013380 con 0x7fc840060280 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc830014e20 con 0x7fc840060280 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.188+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc840107320 con 0x7fc840060280 2024-02-23T00:32:55.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.189+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8300263a0 con 0x7fc840060280 2024-02-23T00:32:55.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.189+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc840045ad0 con 0x7fc840060280 2024-02-23T00:32:55.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.189+0000 7fc8277fe700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 0x7fc8280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.189+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc830040f50 con 0x7fc840060280 2024-02-23T00:32:55.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.189+0000 7fc83e7fc700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 0x7fc8280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.190+0000 7fc83e7fc700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 0x7fc8280323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc8340096a0 tx=0x7fc834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:55.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.194+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc830012330 con 0x7fc840060280 2024-02-23T00:32:55.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.402+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8401081d0 con 0x7fc840060280 2024-02-23T00:32:55.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.402+0000 7fc8277fe700 1 -- 172.21.15.118:0/1360591471 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc83001db70 con 0x7fc840060280 2024-02-23T00:32:55.471 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:55.471 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:55.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 msgr2=0x7fc8280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 0x7fc8280323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc8340096a0 tx=0x7fc834006b40).stop 2024-02-23T00:32:55.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 msgr2=0x7fc8401067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc8300089f0 tx=0x7fc83000b040).stop 2024-02-23T00:32:55.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 shutdown_connections 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc82802ff10 0x7fc8280323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 --2- 172.21.15.118:0/1360591471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc840060280 0x7fc8401067a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 >> 172.21.15.118:0/1360591471 conn(0x7fc8400f0b70 msgr2=0x7fc8400f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 shutdown_connections 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:55.404+0000 7fc845448700 1 -- 172.21.15.118:0/1360591471 wait complete. 2024-02-23T00:32:55.473 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:55.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:55.203586+0000 mgr.smithi090.pxxmlz (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:56.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:55.403326+0000 mon.smithi090 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.118:0/1360591471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:32:57.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:32:57.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:32:57.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:57.203853+0000 mgr.smithi090.pxxmlz (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:32:59.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 -- 172.21.15.118:0/2386981954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 msgr2=0x7f09f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 --2- 172.21.15.118:0/2386981954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f80f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f09e8009fb0 tx=0x7f09e8009b30).stop 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 -- 172.21.15.118:0/2386981954 shutdown_connections 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 --2- 172.21.15.118:0/2386981954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f80f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 -- 172.21.15.118:0/2386981954 >> 172.21.15.118:0/2386981954 conn(0x7f09f80f09a0 msgr2=0x7f09f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 -- 172.21.15.118:0/2386981954 shutdown_connections 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.106+0000 7f09fd601700 1 -- 172.21.15.118:0/2386981954 wait complete. 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.107+0000 7f09fd601700 1 Processor -- start 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.107+0000 7f09fd601700 1 -- start start 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.107+0000 7f09fd601700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:59.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.107+0000 7f09fd601700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8060220 con 0x7f09f80f51a0 2024-02-23T00:32:59.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.107+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:59.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:37626/0 (socket says 172.21.15.118:37626) 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f7fff700 1 -- 172.21.15.118:0/1867981695 learned_addr learned my addr 172.21.15.118:0/1867981695 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f7fff700 1 -- 172.21.15.118:0/1867981695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8005040 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f7fff700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f09e80089f0 tx=0x7f09e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e8009320 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09e8013380 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8060420 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e8014e20 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.108+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f81099a0 con 0x7f09f80f51a0 2024-02-23T00:32:59.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.109+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09e8014420 con 0x7f09f80f51a0 2024-02-23T00:32:59.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.109+0000 7f09f4ff9700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:59.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.109+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8059100 con 0x7f09f80f51a0 2024-02-23T00:32:59.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.109+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09e8040f00 con 0x7f09f80f51a0 2024-02-23T00:32:59.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.109+0000 7f09f77fe700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:59.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.110+0000 7f09f77fe700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 0x7f09e40323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f09e00096a0 tx=0x7f09e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:59.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.114+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09e80095f0 con 0x7f09f80f51a0 2024-02-23T00:32:59.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.319+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f80fad30 con 0x7f09f80f51a0 2024-02-23T00:32:59.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.320+0000 7f09f4ff9700 1 -- 172.21.15.118:0/1867981695 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09e802c090 con 0x7f09f80f51a0 2024-02-23T00:32:59.322 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:32:59.322 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:32:59.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 msgr2=0x7f09e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:59.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 0x7f09e40323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f09e00096a0 tx=0x7f09e0006b40).stop 2024-02-23T00:32:59.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 msgr2=0x7f09f81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:59.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f09e80089f0 tx=0x7f09e800b040).stop 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 shutdown_connections 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 --2- 172.21.15.118:0/1867981695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09f80f51a0 0x7f09f81092f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 >> 172.21.15.118:0/1867981695 conn(0x7f09f80f09a0 msgr2=0x7f09f80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 shutdown_connections 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:32:59.322+0000 7f09fd601700 1 -- 172.21.15.118:0/1867981695 wait complete. 2024-02-23T00:32:59.324 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:32:59.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:32:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:32:59.204164+0000 mgr.smithi090.pxxmlz (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:00.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:32:59.320809+0000 mon.smithi090 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.118:0/1867981695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:01.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:01.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:01.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:01.204524+0000 mgr.smithi090.pxxmlz (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:03.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 -- 172.21.15.118:0/944393276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 msgr2=0x7f2e540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/944393276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e540f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f2e44009fb0 tx=0x7f2e44009b30).stop 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 -- 172.21.15.118:0/944393276 shutdown_connections 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/944393276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e540f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 -- 172.21.15.118:0/944393276 >> 172.21.15.118:0/944393276 conn(0x7f2e540f0b70 msgr2=0x7f2e540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 -- 172.21.15.118:0/944393276 shutdown_connections 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.289+0000 7f2e5be6d700 1 -- 172.21.15.118:0/944393276 wait complete. 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e5be6d700 1 Processor -- start 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e5be6d700 1 -- start start 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e5be6d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e5be6d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e54060220 con 0x7f2e540f5410 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e59c09700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e59c09700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:37650/0 (socket says 172.21.15.118:37650) 2024-02-23T00:33:03.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.290+0000 7f2e59c09700 1 -- 172.21.15.118:0/2633688826 learned_addr learned my addr 172.21.15.118:0/2633688826 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e59c09700 1 -- 172.21.15.118:0/2633688826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e44005040 con 0x7f2e540f5410 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e59c09700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2e44005010 tx=0x7f2e4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e44009320 con 0x7f2e540f5410 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e44013380 con 0x7f2e540f5410 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e54060420 con 0x7f2e540f5410 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e44014e50 con 0x7f2e540f5410 2024-02-23T00:33:03.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.291+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e54109be0 con 0x7f2e540f5410 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.292+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e440095f0 con 0x7f2e540f5410 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.292+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e54059100 con 0x7f2e540f5410 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.292+0000 7f2e4a7fc700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 0x7f2e400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.292+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e44040fa0 con 0x7f2e540f5410 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.292+0000 7f2e59408700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 0x7f2e400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:03.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.293+0000 7f2e59408700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 0x7f2e400323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2e500096a0 tx=0x7f2e50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:03.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.297+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e44014420 con 0x7f2e540f5410 2024-02-23T00:33:03.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.506+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e54045ad0 con 0x7f2e540f5410 2024-02-23T00:33:03.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.506+0000 7f2e4a7fc700 1 -- 172.21.15.118:0/2633688826 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e44010230 con 0x7f2e540f5410 2024-02-23T00:33:03.509 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:03.510 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 msgr2=0x7f2e400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 0x7f2e400323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2e500096a0 tx=0x7f2e50006b40).stop 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 msgr2=0x7f2e54109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2e44005010 tx=0x7f2e4400b040).stop 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 shutdown_connections 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f2e4002ff10 0x7f2e400323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 --2- 172.21.15.118:0/2633688826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e540f5410 0x7f2e54109530 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 >> 172.21.15.118:0/2633688826 conn(0x7f2e540f0b70 msgr2=0x7f2e540f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 shutdown_connections 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:03.508+0000 7f2e5be6d700 1 -- 172.21.15.118:0/2633688826 wait complete. 2024-02-23T00:33:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:03.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:03.204786+0000 mgr.smithi090.pxxmlz (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:04.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:03.508343+0000 mon.smithi090 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.118:0/2633688826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:05.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:05.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:05.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:05.205103+0000 mgr.smithi090.pxxmlz (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 -- 172.21.15.118:0/1996238276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 msgr2=0x7f64080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 --2- 172.21.15.118:0/1996238276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64080f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f63f0009fb0 tx=0x7f63f0009b30).stop 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 -- 172.21.15.118:0/1996238276 shutdown_connections 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 --2- 172.21.15.118:0/1996238276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64080f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 -- 172.21.15.118:0/1996238276 >> 172.21.15.118:0/1996238276 conn(0x7f64080f0b70 msgr2=0x7f64080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 -- 172.21.15.118:0/1996238276 shutdown_connections 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 -- 172.21.15.118:0/1996238276 wait complete. 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.349+0000 7f640dcfb700 1 Processor -- start 2024-02-23T00:33:07.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f640dcfb700 1 -- start start 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f640dcfb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f640dcfb700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408106cd0 con 0x7f64080f5410 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f64077fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f64077fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47752/0 (socket says 172.21.15.118:47752) 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f64077fe700 1 -- 172.21.15.118:0/341416498 learned_addr learned my addr 172.21.15.118:0/341416498 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.350+0000 7f64077fe700 1 -- 172.21.15.118:0/341416498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f0005040 con 0x7f64080f5410 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f64077fe700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63f0009320 con 0x7f64080f5410 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6408106ed0 con 0x7f64080f5410 2024-02-23T00:33:07.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6408107360 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63f0013380 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63f0014df0 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.351+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63f00095f0 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.352+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6408045ad0 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.352+0000 7f63fffff700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 0x7f63f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.352+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63f0040f70 con 0x7f64080f5410 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.352+0000 7f6406ffd700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 0x7f63f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:07.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.352+0000 7f6406ffd700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 0x7f63f40323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f63f80096a0 tx=0x7f63f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:07.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.356+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63f001dbb0 con 0x7f64080f5410 2024-02-23T00:33:07.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.565+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6408108210 con 0x7f64080f5410 2024-02-23T00:33:07.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.566+0000 7f63fffff700 1 -- 172.21.15.118:0/341416498 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63f001d9d0 con 0x7f64080f5410 2024-02-23T00:33:07.568 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:07.568 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 msgr2=0x7f63f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 0x7f63f40323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f63f80096a0 tx=0x7f63f8006b40).stop 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 msgr2=0x7f64081067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f000b040).stop 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 shutdown_connections 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f63f402ff10 0x7f63f40323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 --2- 172.21.15.118:0/341416498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64080f5410 0x7f64081067e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 >> 172.21.15.118:0/341416498 conn(0x7f64080f0b70 msgr2=0x7f64080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 shutdown_connections 2024-02-23T00:33:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:07.567+0000 7f640dcfb700 1 -- 172.21.15.118:0/341416498 wait complete. 2024-02-23T00:33:07.571 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:07.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:07.205518+0000 mgr.smithi090.pxxmlz (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:08.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:07.567527+0000 mon.smithi090 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.118:0/341416498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:09.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:09.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:09.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:09.205776+0000 mgr.smithi090.pxxmlz (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 -- 172.21.15.118:0/394823362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 msgr2=0x7effe40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 --2- 172.21.15.118:0/394823362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe40f9190 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7effd8009fb0 tx=0x7effd8009b30).stop 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 -- 172.21.15.118:0/394823362 shutdown_connections 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 --2- 172.21.15.118:0/394823362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe40f9190 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 -- 172.21.15.118:0/394823362 >> 172.21.15.118:0/394823362 conn(0x7effe40f0b70 msgr2=0x7effe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 -- 172.21.15.118:0/394823362 shutdown_connections 2024-02-23T00:33:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.415+0000 7effe92e7700 1 -- 172.21.15.118:0/394823362 wait complete. 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe92e7700 1 Processor -- start 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe92e7700 1 -- start start 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe92e7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe92e7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe4106cc0 con 0x7effe40601e0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:47756/0 (socket says 172.21.15.118:47756) 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.416+0000 7effe2ffd700 1 -- 172.21.15.118:0/3525999379 learned_addr learned my addr 172.21.15.118:0/3525999379 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effe2ffd700 1 -- 172.21.15.118:0/3525999379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd8005040 con 0x7effe40601e0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effe2ffd700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7effd8007a10 tx=0x7effd8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effd8018070 con 0x7effe40601e0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effd8013410 con 0x7effe40601e0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe4106ec0 con 0x7effe40601e0 2024-02-23T00:33:11.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effd801c3c0 con 0x7effe40601e0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.417+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe4107350 con 0x7effe40601e0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effd801d420 con 0x7effe40601e0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe4045ad0 con 0x7effe40601e0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effcb7fe700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 0x7effcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effd8041480 con 0x7effe40601e0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effe27fc700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 0x7effcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.418+0000 7effe27fc700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 0x7effcc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7effd40096a0 tx=0x7effd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:11.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.423+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effd8013ca0 con 0x7effe40601e0 2024-02-23T00:33:11.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.629+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effe4108200 con 0x7effe40601e0 2024-02-23T00:33:11.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.630+0000 7effcb7fe700 1 -- 172.21.15.118:0/3525999379 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effd801b370 con 0x7effe40601e0 2024-02-23T00:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:11.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 msgr2=0x7effcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:11.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 0x7effcc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7effd40096a0 tx=0x7effd4006b40).stop 2024-02-23T00:33:11.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 msgr2=0x7effe41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:11.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7effd8007a10 tx=0x7effd8009b30).stop 2024-02-23T00:33:11.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 shutdown_connections 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7effcc02ff60 0x7effcc032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 --2- 172.21.15.118:0/3525999379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7effe40601e0 0x7effe41067d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 >> 172.21.15.118:0/3525999379 conn(0x7effe40f0b70 msgr2=0x7effe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 shutdown_connections 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:11.631+0000 7effe92e7700 1 -- 172.21.15.118:0/3525999379 wait complete. 2024-02-23T00:33:11.636 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:11.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:11.206166+0000 mgr.smithi090.pxxmlz (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:12.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:11.631637+0000 mon.smithi090 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.118:0/3525999379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:13.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:13.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:13.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:13.206563+0000 mgr.smithi090.pxxmlz (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:15.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 -- 172.21.15.118:0/3734334502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 msgr2=0x7fd4700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:15.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 --2- 172.21.15.118:0/3734334502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fd458009fb0 tx=0x7fd458009b30).stop 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 -- 172.21.15.118:0/3734334502 shutdown_connections 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 --2- 172.21.15.118:0/3734334502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 -- 172.21.15.118:0/3734334502 >> 172.21.15.118:0/3734334502 conn(0x7fd4700f0b70 msgr2=0x7fd4700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 -- 172.21.15.118:0/3734334502 shutdown_connections 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.224+0000 7fd474f55700 1 -- 172.21.15.118:0/3734334502 wait complete. 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd474f55700 1 Processor -- start 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd474f55700 1 -- start start 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd474f55700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd474f55700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700fb960 con 0x7fd4700f5410 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd46e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd46e59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60030/0 (socket says 172.21.15.118:60030) 2024-02-23T00:33:15.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd46e59c700 1 -- 172.21.15.118:0/2745561500 learned_addr learned my addr 172.21.15.118:0/2745561500 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.225+0000 7fd46e59c700 1 -- 172.21.15.118:0/2745561500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd458005040 con 0x7fd4700f5410 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd46e59c700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fd458005010 tx=0x7fd45800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd458009320 con 0x7fd4700f5410 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4700fbb00 con 0x7fd4700f5410 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd458013380 con 0x7fd4700f5410 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd458014e50 con 0x7fd4700f5410 2024-02-23T00:33:15.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.226+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4700fbf90 con 0x7fd4700f5410 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4580095f0 con 0x7fd4700f5410 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd470059100 con 0x7fd4700f5410 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd466ffd700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 0x7fd45c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd458040f80 con 0x7fd4700f5410 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd46dd9b700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 0x7fd45c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:15.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.227+0000 7fd46dd9b700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 0x7fd45c0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd4600096a0 tx=0x7fd460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:15.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.231+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd458014420 con 0x7fd4700f5410 2024-02-23T00:33:15.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.435+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd470045ad0 con 0x7fd4700f5410 2024-02-23T00:33:15.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.436+0000 7fd466ffd700 1 -- 172.21.15.118:0/2745561500 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd458010230 con 0x7fd4700f5410 2024-02-23T00:33:15.438 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:15.438 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:15.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.437+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 msgr2=0x7fd45c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.437+0000 7fd474f55700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 0x7fd45c0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd4600096a0 tx=0x7fd460006b40).stop 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.437+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 msgr2=0x7fd4700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.437+0000 7fd474f55700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fd458005010 tx=0x7fd45800b040).stop 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 shutdown_connections 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd45c02ff10 0x7fd45c0323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 --2- 172.21.15.118:0/2745561500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700f5410 0x7fd4700fd4e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 >> 172.21.15.118:0/2745561500 conn(0x7fd4700f0b70 msgr2=0x7fd47005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 shutdown_connections 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:15.438+0000 7fd474f55700 1 -- 172.21.15.118:0/2745561500 wait complete. 2024-02-23T00:33:15.440 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:15.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:15.206989+0000 mgr.smithi090.pxxmlz (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:16.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:15.436882+0000 mon.smithi090 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.118:0/2745561500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:17.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:17.071 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:17.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:17.207321+0000 mgr.smithi090.pxxmlz (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:19.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 -- 172.21.15.118:0/3779019103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 msgr2=0x7ff7d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 --2- 172.21.15.118:0/3779019103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d00f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7ff7c0009fb0 tx=0x7ff7c0009b30).stop 2024-02-23T00:33:19.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 -- 172.21.15.118:0/3779019103 shutdown_connections 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 --2- 172.21.15.118:0/3779019103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d00f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 -- 172.21.15.118:0/3779019103 >> 172.21.15.118:0/3779019103 conn(0x7ff7d00f0b70 msgr2=0x7ff7d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 -- 172.21.15.118:0/3779019103 shutdown_connections 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.357+0000 7ff7d815e700 1 -- 172.21.15.118:0/3779019103 wait complete. 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d815e700 1 Processor -- start 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d815e700 1 -- start start 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d815e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d815e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0105c00 con 0x7ff7d00f5410 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d5efa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d5efa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60044/0 (socket says 172.21.15.118:60044) 2024-02-23T00:33:19.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.358+0000 7ff7d5efa700 1 -- 172.21.15.118:0/1747655054 learned_addr learned my addr 172.21.15.118:0/1747655054 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7d5efa700 1 -- 172.21.15.118:0/1747655054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0005040 con 0x7ff7d00f5410 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7d5efa700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7ff7c0005010 tx=0x7ff7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0009320 con 0x7ff7d00f5410 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d0105e00 con 0x7ff7d00f5410 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7c0013380 con 0x7ff7d00f5410 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0014e50 con 0x7ff7d00f5410 2024-02-23T00:33:19.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.359+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d0106290 con 0x7ff7d00f5410 2024-02-23T00:33:19.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.360+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7c00095f0 con 0x7ff7d00f5410 2024-02-23T00:33:19.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.360+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0045ad0 con 0x7ff7d00f5410 2024-02-23T00:33:19.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.360+0000 7ff7c67fc700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 0x7ff7bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.360+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7c0040ef0 con 0x7ff7d00f5410 2024-02-23T00:33:19.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.360+0000 7ff7d56f9700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 0x7ff7bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.361+0000 7ff7d56f9700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 0x7ff7bc0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff7cc0096a0 tx=0x7ff7cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:19.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.365+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7c0014420 con 0x7ff7d00f5410 2024-02-23T00:33:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.568+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7d01070e0 con 0x7ff7d00f5410 2024-02-23T00:33:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.569+0000 7ff7c67fc700 1 -- 172.21.15.118:0/1747655054 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7c0010230 con 0x7ff7d00f5410 2024-02-23T00:33:19.570 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:19.570 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 msgr2=0x7ff7bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 0x7ff7bc0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff7cc0096a0 tx=0x7ff7cc006b40).stop 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 msgr2=0x7ff7d0105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7ff7c0005010 tx=0x7ff7c000b040).stop 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 shutdown_connections 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff7bc02ff10 0x7ff7bc0323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 --2- 172.21.15.118:0/1747655054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d00f5410 0x7ff7d0105710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 >> 172.21.15.118:0/1747655054 conn(0x7ff7d00f0b70 msgr2=0x7ff7d00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 shutdown_connections 2024-02-23T00:33:19.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:19.570+0000 7ff7d815e700 1 -- 172.21.15.118:0/1747655054 wait complete. 2024-02-23T00:33:19.573 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:19.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:19.207618+0000 mgr.smithi090.pxxmlz (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:20.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:19.569694+0000 mon.smithi090 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.118:0/1747655054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:21.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:21.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:21.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:21.207976+0000 mgr.smithi090.pxxmlz (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 -- 172.21.15.118:0/612671934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 msgr2=0x7f35880f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 --2- 172.21.15.118:0/612671934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35880f9150 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f357c009fb0 tx=0x7f357c009b30).stop 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 -- 172.21.15.118:0/612671934 shutdown_connections 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 --2- 172.21.15.118:0/612671934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35880f9150 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 -- 172.21.15.118:0/612671934 >> 172.21.15.118:0/612671934 conn(0x7f35880f0b70 msgr2=0x7f35880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:23.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 -- 172.21.15.118:0/612671934 shutdown_connections 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 -- 172.21.15.118:0/612671934 wait complete. 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.476+0000 7f358f07d700 1 Processor -- start 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358f07d700 1 -- start start 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358f07d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358f07d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3588106c90 con 0x7f3588060220 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358ce19700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358ce19700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60062/0 (socket says 172.21.15.118:60062) 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.477+0000 7f358ce19700 1 -- 172.21.15.118:0/3818796275 learned_addr learned my addr 172.21.15.118:0/3818796275 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f358ce19700 1 -- 172.21.15.118:0/3818796275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c005040 con 0x7f3588060220 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f358ce19700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f357c0089f0 tx=0x7f357c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357c009320 con 0x7f3588060220 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3588106e90 con 0x7f3588060220 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f357c013380 con 0x7f3588060220 2024-02-23T00:33:23.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357c014e20 con 0x7f3588060220 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.478+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3588107320 con 0x7f3588060220 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.479+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f357c0095f0 con 0x7f3588060220 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.479+0000 7f35857fa700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 0x7f35780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.479+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3588045ad0 con 0x7f3588060220 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.479+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f357c040e90 con 0x7f3588060220 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.479+0000 7f3587fff700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 0x7f35780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.480+0000 7f3587fff700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 0x7f35780323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f35740096a0 tx=0x7f3574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:23.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.484+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f357c01db60 con 0x7f3588060220 2024-02-23T00:33:23.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.690+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35881081d0 con 0x7f3588060220 2024-02-23T00:33:23.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.691+0000 7f35857fa700 1 -- 172.21.15.118:0/3818796275 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f357c01dd70 con 0x7f3588060220 2024-02-23T00:33:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:23.693 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 msgr2=0x7f35780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 0x7f35780323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f35740096a0 tx=0x7f3574006b40).stop 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 msgr2=0x7f35881067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f357c0089f0 tx=0x7f357c00b040).stop 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 shutdown_connections 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f357802ff10 0x7f35780323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 --2- 172.21.15.118:0/3818796275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3588060220 0x7f35881067a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 >> 172.21.15.118:0/3818796275 conn(0x7f35880f0b70 msgr2=0x7f35880f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 shutdown_connections 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:23.693+0000 7f358f07d700 1 -- 172.21.15.118:0/3818796275 wait complete. 2024-02-23T00:33:23.695 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:23.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:23.208255+0000 mgr.smithi090.pxxmlz (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:23.692028+0000 mon.smithi090 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.118:0/3818796275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:33:23.945602+0000 mgr.smithi090.pxxmlz (mgr.14162) 308 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi090 ... 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:31:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:32:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:32:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:33:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('uid/gid not found') 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: uid/gid not found 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:33:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: yield (conn, connr) 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: code, '\n'.join(err))) 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi090 ... 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi090 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:31:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:32:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr time="2024-02-23T00:32:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: Traceback (most recent call last): 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-23T00:33:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: main() 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: r = ctx.func(ctx) 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: return func(ctx) 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: File "/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: raise RuntimeError('uid/gid not found') 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: RuntimeError: uid/gid not found 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:23.946274+0000 mon.smithi090 (mon.0) 283 : audit [DBG] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:33:24.754 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:24.171833+0000 mon.smithi090 (mon.0) 284 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:25.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:25.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:25.772 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:24.505800+0000 mon.smithi090 (mon.0) 285 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:24.737203+0000 mon.smithi090 (mon.0) 286 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:25.208484+0000 mgr.smithi090.pxxmlz (mgr.14162) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:25.252617+0000 mon.smithi090 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:33:25.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:25.259326+0000 mon.smithi090 (mon.0) 288 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:33:27.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:26.024861+0000 mon.smithi090 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:28.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 -- 172.21.15.118:0/1533997592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 msgr2=0x7eff980fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 --2- 172.21.15.118:0/1533997592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff980fb550 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7eff88009fb0 tx=0x7eff88009b30).stop 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 -- 172.21.15.118:0/1533997592 shutdown_connections 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 --2- 172.21.15.118:0/1533997592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff980fb550 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 -- 172.21.15.118:0/1533997592 >> 172.21.15.118:0/1533997592 conn(0x7eff980f0b70 msgr2=0x7eff980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 -- 172.21.15.118:0/1533997592 shutdown_connections 2024-02-23T00:33:28.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.160+0000 7eff9d06f700 1 -- 172.21.15.118:0/1533997592 wait complete. 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff9d06f700 1 Processor -- start 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff9d06f700 1 -- start start 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff9d06f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff9d06f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff98108e90 con 0x7eff98060220 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff96d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:28.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff96d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53386/0 (socket says 172.21.15.118:53386) 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff96d9d700 1 -- 172.21.15.118:0/2659345998 learned_addr learned my addr 172.21.15.118:0/2659345998 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff96d9d700 1 -- 172.21.15.118:0/2659345998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff88005040 con 0x7eff98060220 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff96d9d700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7eff880089f0 tx=0x7eff8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff88009320 con 0x7eff98060220 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff88013380 con 0x7eff98060220 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff88014e20 con 0x7eff98060220 2024-02-23T00:33:28.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.161+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff98109090 con 0x7eff98060220 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.162+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff98109520 con 0x7eff98060220 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.162+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff880263a0 con 0x7eff98060220 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.163+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff98045ad0 con 0x7eff98060220 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.163+0000 7eff7f7fe700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 0x7eff800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.163+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff88040fa0 con 0x7eff98060220 2024-02-23T00:33:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.163+0000 7eff9659c700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 0x7eff800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:28.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.163+0000 7eff9659c700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 0x7eff800323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7eff8c0096a0 tx=0x7eff8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:28.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.168+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff88026760 con 0x7eff98060220 2024-02-23T00:33:28.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.385+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff9810a3d0 con 0x7eff98060220 2024-02-23T00:33:28.387 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.386+0000 7eff7f7fe700 1 -- 172.21.15.118:0/2659345998 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff8801db70 con 0x7eff98060220 2024-02-23T00:33:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:28.387 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 msgr2=0x7eff800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 0x7eff800323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7eff8c0096a0 tx=0x7eff8c006b40).stop 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 msgr2=0x7eff981089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7eff880089f0 tx=0x7eff8800b040).stop 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 shutdown_connections 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7eff8002ff10 0x7eff800323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 --2- 172.21.15.118:0/2659345998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff98060220 0x7eff981089a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 >> 172.21.15.118:0/2659345998 conn(0x7eff980f0b70 msgr2=0x7eff980f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 shutdown_connections 2024-02-23T00:33:28.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:28.387+0000 7eff9d06f700 1 -- 172.21.15.118:0/2659345998 wait complete. 2024-02-23T00:33:28.390 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:28.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:27.208762+0000 mgr.smithi090.pxxmlz (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:29.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:28.386587+0000 mon.smithi090 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.118:0/2659345998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:30.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:30.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:30.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:29.209071+0000 mgr.smithi090.pxxmlz (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:31.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:30.062139+0000 mon.smithi090 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:31.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:30.340137+0000 mon.smithi090 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:31.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:30.342883+0000 mon.smithi090 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 -- 172.21.15.118:0/3512317959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 msgr2=0x7f39940f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 --2- 172.21.15.118:0/3512317959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39940f9190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f3984009fb0 tx=0x7f3984009b30).stop 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 -- 172.21.15.118:0/3512317959 shutdown_connections 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 --2- 172.21.15.118:0/3512317959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39940f9190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 -- 172.21.15.118:0/3512317959 >> 172.21.15.118:0/3512317959 conn(0x7f39940f0b70 msgr2=0x7f39940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 -- 172.21.15.118:0/3512317959 shutdown_connections 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 -- 172.21.15.118:0/3512317959 wait complete. 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.435+0000 7f399c6a4700 1 Processor -- start 2024-02-23T00:33:32.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399c6a4700 1 -- start start 2024-02-23T00:33:32.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399c6a4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:32.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399c6a4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3994108eb0 con 0x7f39940601e0 2024-02-23T00:33:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399a440700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399a440700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53398/0 (socket says 172.21.15.118:53398) 2024-02-23T00:33:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399a440700 1 -- 172.21.15.118:0/1610479521 learned_addr learned my addr 172.21.15.118:0/1610479521 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399a440700 1 -- 172.21.15.118:0/1610479521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3984005040 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399a440700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3984009830 tx=0x7f3984009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3984018070 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.436+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39941090b0 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3984013410 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398401c3c0 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3994109540 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f398401d420 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3994045ad0 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.437+0000 7f398affd700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 0x7f3980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.438+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3984041500 con 0x7f39940601e0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.438+0000 7f3999c3f700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 0x7f3980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:32.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.438+0000 7f3999c3f700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 0x7f3980032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f39900096a0 tx=0x7f3990006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:32.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.442+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3984013ca0 con 0x7f39940601e0 2024-02-23T00:33:32.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:30.340860+0000 mgr.smithi090.pxxmlz (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:32.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cephadm 2024-02-23T00:33:30.345023+0000 mgr.smithi090.pxxmlz (mgr.14162) 313 : cephadm [INF] Deploying daemon alertmanager.smithi090 on smithi090 2024-02-23T00:33:32.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:31.338766+0000 mon.smithi090 (mon.0) 294 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-23T00:33:32.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.648+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f399410a390 con 0x7f39940601e0 2024-02-23T00:33:32.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.649+0000 7f398affd700 1 -- 172.21.15.118:0/1610479521 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f398401b370 con 0x7f39940601e0 2024-02-23T00:33:32.650 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:32.650 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 msgr2=0x7f3980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 0x7f3980032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f39900096a0 tx=0x7f3990006b40).stop 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 msgr2=0x7f39941089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3984009830 tx=0x7f3984009b30).stop 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 shutdown_connections 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f398002ff60 0x7f3980032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 --2- 172.21.15.118:0/1610479521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39940601e0 0x7f39941089c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 >> 172.21.15.118:0/1610479521 conn(0x7f39940f0b70 msgr2=0x7f39940f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 shutdown_connections 2024-02-23T00:33:32.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:32.650+0000 7f399c6a4700 1 -- 172.21.15.118:0/1610479521 wait complete. 2024-02-23T00:33:32.653 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:33.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:32.649462+0000 mon.smithi090 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.118:0/1610479521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:34.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:34.492 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:34.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:34 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:32.341210+0000 mgr.smithi090.pxxmlz (mgr.14162) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:36.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:34.341530+0000 mgr.smithi090.pxxmlz (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.755+0000 7f01d392e700 1 -- 172.21.15.118:0/3268089574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 msgr2=0x7f01cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.755+0000 7f01d392e700 1 --2- 172.21.15.118:0/3268089574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f01c8009fb0 tx=0x7f01c8009b30).stop 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- 172.21.15.118:0/3268089574 shutdown_connections 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 --2- 172.21.15.118:0/3268089574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- 172.21.15.118:0/3268089574 >> 172.21.15.118:0/3268089574 conn(0x7f01cc0f0b70 msgr2=0x7f01cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- 172.21.15.118:0/3268089574 shutdown_connections 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- 172.21.15.118:0/3268089574 wait complete. 2024-02-23T00:33:36.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 Processor -- start 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- start start 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.756+0000 7f01d392e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc105bd0 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d16ca700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d16ca700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:56734/0 (socket says 172.21.15.118:56734) 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d16ca700 1 -- 172.21.15.118:0/933755376 learned_addr learned my addr 172.21.15.118:0/933755376 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d16ca700 1 -- 172.21.15.118:0/933755376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8005040 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d16ca700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f01c8005010 tx=0x7f01c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c8009320 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc105dd0 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01c8013380 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c8014e50 con 0x7f01cc0f5410 2024-02-23T00:33:36.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.757+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc106260 con 0x7f01cc0f5410 2024-02-23T00:33:36.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.758+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01c80095f0 con 0x7f01cc0f5410 2024-02-23T00:33:36.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.758+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc045ad0 con 0x7f01cc0f5410 2024-02-23T00:33:36.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.758+0000 7f01bdffb700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:36.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.758+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01c8040f80 con 0x7f01cc0f5410 2024-02-23T00:33:36.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.759+0000 7f01d0ec9700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:36.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.759+0000 7f01d0ec9700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 0x7f01b80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f01c00096a0 tx=0x7f01c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:36.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.763+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01c8014420 con 0x7f01cc0f5410 2024-02-23T00:33:36.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.969+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01cc107110 con 0x7f01cc0f5410 2024-02-23T00:33:36.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.970+0000 7f01bdffb700 1 -- 172.21.15.118:0/933755376 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01c8010230 con 0x7f01cc0f5410 2024-02-23T00:33:36.972 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:36.972 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:36.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 msgr2=0x7f01b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:36.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 0x7f01b80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f01c00096a0 tx=0x7f01c0006b40).stop 2024-02-23T00:33:36.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 msgr2=0x7f01cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:36.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f01c8005010 tx=0x7f01c800b040).stop 2024-02-23T00:33:36.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 shutdown_connections 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 --2- 172.21.15.118:0/933755376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01cc0f5410 0x7f01cc1056e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 >> 172.21.15.118:0/933755376 conn(0x7f01cc0f0b70 msgr2=0x7f01cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 shutdown_connections 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:36.972+0000 7f01d392e700 1 -- 172.21.15.118:0/933755376 wait complete. 2024-02-23T00:33:36.974 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:37.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:36.970957+0000 mon.smithi090 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.118:0/933755376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:38.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:38 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:36.341813+0000 mgr.smithi090.pxxmlz (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:38.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:38.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:40.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:38.342084+0000 mgr.smithi090.pxxmlz (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:40.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 -- 172.21.15.118:0/1746720496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 msgr2=0x7f67800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:40.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 --2- 172.21.15.118:0/1746720496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67800f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f6774009fb0 tx=0x7f6774009b30).stop 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 -- 172.21.15.118:0/1746720496 shutdown_connections 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 --2- 172.21.15.118:0/1746720496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67800f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 -- 172.21.15.118:0/1746720496 >> 172.21.15.118:0/1746720496 conn(0x7f67800f0b70 msgr2=0x7f67800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 -- 172.21.15.118:0/1746720496 shutdown_connections 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 -- 172.21.15.118:0/1746720496 wait complete. 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.924+0000 7f678858b700 1 Processor -- start 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f678858b700 1 -- start start 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f678858b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f678858b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6780106cd0 con 0x7f67800f5410 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6786327700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6786327700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:56752/0 (socket says 172.21.15.118:56752) 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6786327700 1 -- 172.21.15.118:0/1783846846 learned_addr learned my addr 172.21.15.118:0/1783846846 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:40.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6786327700 1 -- 172.21.15.118:0/1783846846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774005040 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6786327700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6774005010 tx=0x7f677400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.925+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774009320 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6780106ed0 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6774013380 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774014e50 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6780107360 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67740095f0 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6780045ad0 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.926+0000 7f6772ffd700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 0x7f676c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.927+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6774040fa0 con 0x7f67800f5410 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.927+0000 7f6785b26700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 0x7f676c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.927+0000 7f6785b26700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 0x7f676c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:40.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:40.931+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6774014420 con 0x7f67800f5410 2024-02-23T00:33:41.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.138+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67800fafa0 con 0x7f67800f5410 2024-02-23T00:33:41.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.138+0000 7f6772ffd700 1 -- 172.21.15.118:0/1783846846 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6774010230 con 0x7f67800f5410 2024-02-23T00:33:41.140 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:41.140 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:41.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 msgr2=0x7f676c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 0x7f676c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).stop 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 msgr2=0x7f67801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6774005010 tx=0x7f677400b040).stop 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 shutdown_connections 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f676c02ff10 0x7f676c0323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 --2- 172.21.15.118:0/1783846846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67800f5410 0x7f67801067e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 >> 172.21.15.118:0/1783846846 conn(0x7f67800f0b70 msgr2=0x7f678005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 shutdown_connections 2024-02-23T00:33:41.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:41.140+0000 7f678858b700 1 -- 172.21.15.118:0/1783846846 wait complete. 2024-02-23T00:33:41.143 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:42.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:40.342349+0000 mgr.smithi090.pxxmlz (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:42.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:42 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:41.139154+0000 mon.smithi090 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.118:0/1783846846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:42.924 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:42.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:44.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:42.342663+0000 mgr.smithi090.pxxmlz (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:44.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 -- 172.21.15.118:0/4240520219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 msgr2=0x7f0a040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:44.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 --2- 172.21.15.118:0/4240520219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a040f55c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f09f8009fb0 tx=0x7f09f8009b30).stop 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 -- 172.21.15.118:0/4240520219 shutdown_connections 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 --2- 172.21.15.118:0/4240520219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a040f55c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 -- 172.21.15.118:0/4240520219 >> 172.21.15.118:0/4240520219 conn(0x7f0a040f09a0 msgr2=0x7f0a040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 -- 172.21.15.118:0/4240520219 shutdown_connections 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.993+0000 7f0a0994a700 1 -- 172.21.15.118:0/4240520219 wait complete. 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a0994a700 1 Processor -- start 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a0994a700 1 -- start start 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a0994a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a0994a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04060280 con 0x7f0a040f51a0 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a08948700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a08948700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50814/0 (socket says 172.21.15.118:50814) 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.994+0000 7f0a08948700 1 -- 172.21.15.118:0/707867675 learned_addr learned my addr 172.21.15.118:0/707867675 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a08948700 1 -- 172.21.15.118:0/707867675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8005040 con 0x7f0a040f51a0 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a08948700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f09f80089f0 tx=0x7f09f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:44.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09f8009320 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04060480 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09f8013380 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09f8014e20 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.995+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a041099c0 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.996+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09f80095f0 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.996+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04045ad0 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.996+0000 7f0a017fa700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 0x7f09f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.996+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09f8040f50 con 0x7f0a040f51a0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.996+0000 7f0a03fff700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 0x7f09f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:44.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:44.997+0000 7f0a03fff700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 0x7f09f40323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f09f00096a0 tx=0x7f09f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:45.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.001+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09f8014420 con 0x7f0a040f51a0 2024-02-23T00:33:45.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.205+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a040fad30 con 0x7f0a040f51a0 2024-02-23T00:33:45.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.206+0000 7f0a017fa700 1 -- 172.21.15.118:0/707867675 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09f801d980 con 0x7f0a040f51a0 2024-02-23T00:33:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:45.208 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 msgr2=0x7f09f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 0x7f09f40323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f09f00096a0 tx=0x7f09f0006b40).stop 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 msgr2=0x7f0a04109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f09f80089f0 tx=0x7f09f800b040).stop 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 shutdown_connections 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f09f402ff10 0x7f09f40323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:45.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 --2- 172.21.15.118:0/707867675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a040f51a0 0x7f0a04109290 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 >> 172.21.15.118:0/707867675 conn(0x7f0a040f09a0 msgr2=0x7f0a040f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 shutdown_connections 2024-02-23T00:33:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:45.208+0000 7f0a0994a700 1 -- 172.21.15.118:0/707867675 wait complete. 2024-02-23T00:33:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:46.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:44.342957+0000 mgr.smithi090.pxxmlz (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:46.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:46 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:45.207032+0000 mon.smithi090 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.118:0/707867675' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:47.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:47.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:48.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:48 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:46.343191+0000 mgr.smithi090.pxxmlz (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- 172.21.15.118:0/785715485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 msgr2=0x7fde680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 --2- 172.21.15.118:0/785715485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde680f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fde58009fb0 tx=0x7fde58009b30).stop 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- 172.21.15.118:0/785715485 shutdown_connections 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 --2- 172.21.15.118:0/785715485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde680f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- 172.21.15.118:0/785715485 >> 172.21.15.118:0/785715485 conn(0x7fde680f09a0 msgr2=0x7fde680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- 172.21.15.118:0/785715485 shutdown_connections 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- 172.21.15.118:0/785715485 wait complete. 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 Processor -- start 2024-02-23T00:33:49.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.205+0000 7fde6759e700 1 -- start start 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6759e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6759e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681059c0 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50836/0 (socket says 172.21.15.118:50836) 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6659c700 1 -- 172.21.15.118:0/2951034689 learned_addr learned my addr 172.21.15.118:0/2951034689 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6659c700 1 -- 172.21.15.118:0/2951034689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58005040 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6659c700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fde580089f0 tx=0x7fde5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde58009320 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68105bc0 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde58013380 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde58014e20 con 0x7fde680f51a0 2024-02-23T00:33:49.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.206+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68106050 con 0x7fde680f51a0 2024-02-23T00:33:49.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.207+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde580095f0 con 0x7fde680f51a0 2024-02-23T00:33:49.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.207+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68045ad0 con 0x7fde680f51a0 2024-02-23T00:33:49.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.207+0000 7fde56ffd700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:49.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.207+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde58040f50 con 0x7fde680f51a0 2024-02-23T00:33:49.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.208+0000 7fde65d9b700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:49.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.208+0000 7fde65d9b700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 0x7fde500323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:49.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.212+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde58014420 con 0x7fde680f51a0 2024-02-23T00:33:49.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.421+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde68106f00 con 0x7fde680f51a0 2024-02-23T00:33:49.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.421+0000 7fde56ffd700 1 -- 172.21.15.118:0/2951034689 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde5801d980 con 0x7fde680f51a0 2024-02-23T00:33:49.424 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:49.425 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 msgr2=0x7fde500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 0x7fde500323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fde5c0096a0 tx=0x7fde5c006b40).stop 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 msgr2=0x7fde681054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fde580089f0 tx=0x7fde5800b040).stop 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 shutdown_connections 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 --2- 172.21.15.118:0/2951034689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde680f51a0 0x7fde681054d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 >> 172.21.15.118:0/2951034689 conn(0x7fde680f09a0 msgr2=0x7fde680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 shutdown_connections 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:49.423+0000 7fde6759e700 1 -- 172.21.15.118:0/2951034689 wait complete. 2024-02-23T00:33:49.426 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:50.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:48.343531+0000 mgr.smithi090.pxxmlz (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:50.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:50 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:49.423273+0000 mon.smithi090 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.118:0/2951034689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:51.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:51.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:52.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:52 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:50.343908+0000 mgr.smithi090.pxxmlz (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:53.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- 172.21.15.118:0/3265679757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 msgr2=0x7ff69c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 --2- 172.21.15.118:0/3265679757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c0f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ff690009fb0 tx=0x7ff690009b30).stop 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- 172.21.15.118:0/3265679757 shutdown_connections 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 --2- 172.21.15.118:0/3265679757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c0f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- 172.21.15.118:0/3265679757 >> 172.21.15.118:0/3265679757 conn(0x7ff69c0f0b70 msgr2=0x7ff69c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- 172.21.15.118:0/3265679757 shutdown_connections 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- 172.21.15.118:0/3265679757 wait complete. 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 Processor -- start 2024-02-23T00:33:53.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- start start 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.299+0000 7ff6a358a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c060240 con 0x7ff69c0f5410 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a1326700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a1326700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50850/0 (socket says 172.21.15.118:50850) 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a1326700 1 -- 172.21.15.118:0/1470050398 learned_addr learned my addr 172.21.15.118:0/1470050398 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a1326700 1 -- 172.21.15.118:0/1470050398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690005040 con 0x7ff69c0f5410 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a1326700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff690005010 tx=0x7ff69000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff690009320 con 0x7ff69c0f5410 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c060440 con 0x7ff69c0f5410 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff690013380 con 0x7ff69c0f5410 2024-02-23T00:33:53.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff690014e50 con 0x7ff69c0f5410 2024-02-23T00:33:53.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.300+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c109b90 con 0x7ff69c0f5410 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.301+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6900095f0 con 0x7ff69c0f5410 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.301+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c045ad0 con 0x7ff69c0f5410 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.301+0000 7ff68dffb700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 0x7ff6880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.301+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff690040fa0 con 0x7ff69c0f5410 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.302+0000 7ff6a0b25700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 0x7ff6880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:53.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.302+0000 7ff6a0b25700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 0x7ff6880323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff6980096a0 tx=0x7ff698006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:53.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.306+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff690014420 con 0x7ff69c0f5410 2024-02-23T00:33:53.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.509+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff69c10a3b0 con 0x7ff69c0f5410 2024-02-23T00:33:53.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.510+0000 7ff68dffb700 1 -- 172.21.15.118:0/1470050398 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff690010230 con 0x7ff69c0f5410 2024-02-23T00:33:53.512 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:53.512 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:53.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 msgr2=0x7ff6880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:53.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 0x7ff6880323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff6980096a0 tx=0x7ff698006b40).stop 2024-02-23T00:33:53.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 msgr2=0x7ff69c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff690005010 tx=0x7ff69000b040).stop 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 shutdown_connections 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff68802ff10 0x7ff6880323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 --2- 172.21.15.118:0/1470050398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff69c0f5410 0x7ff69c1094e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 >> 172.21.15.118:0/1470050398 conn(0x7ff69c0f0b70 msgr2=0x7ff69c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 shutdown_connections 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:53.511+0000 7ff6a358a700 1 -- 172.21.15.118:0/1470050398 wait complete. 2024-02-23T00:33:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:54.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:52.344245+0000 mgr.smithi090.pxxmlz (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:54.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:54 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:53.511564+0000 mon.smithi090 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.118:0/1470050398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:55.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:55.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:33:56.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:54.344604+0000 mgr.smithi090.pxxmlz (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 -- 172.21.15.118:0/1855352778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 msgr2=0x7f3ba40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 --2- 172.21.15.118:0/1855352778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba40f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f3b94009fb0 tx=0x7f3b94009b30).stop 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 -- 172.21.15.118:0/1855352778 shutdown_connections 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 --2- 172.21.15.118:0/1855352778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba40f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 -- 172.21.15.118:0/1855352778 >> 172.21.15.118:0/1855352778 conn(0x7f3ba40f0b70 msgr2=0x7f3ba40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:57.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 -- 172.21.15.118:0/1855352778 shutdown_connections 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.448+0000 7f3baa124700 1 -- 172.21.15.118:0/1855352778 wait complete. 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3baa124700 1 Processor -- start 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3baa124700 1 -- start start 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3baa124700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3baa124700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4060240 con 0x7f3ba40f5410 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3ba37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3ba37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41346/0 (socket says 172.21.15.118:41346) 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.449+0000 7f3ba37fe700 1 -- 172.21.15.118:0/2924340120 learned_addr learned my addr 172.21.15.118:0/2924340120 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3ba37fe700 1 -- 172.21.15.118:0/2924340120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94005040 con 0x7f3ba40f5410 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3ba37fe700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f3b94005010 tx=0x7f3b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b94009320 con 0x7f3ba40f5410 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba4060440 con 0x7f3ba40f5410 2024-02-23T00:33:57.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b94013380 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b94014e50 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.450+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba4109b90 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.451+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b94014420 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.451+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba4045ad0 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.451+0000 7f3b8bfff700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 0x7f3b8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.451+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b94040f40 con 0x7f3ba40f5410 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.451+0000 7f3ba2ffd700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 0x7f3b8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.452+0000 7f3ba2ffd700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 0x7f3b8c0323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3b980096a0 tx=0x7f3b98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:57.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.456+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b940095f0 con 0x7f3ba40f5410 2024-02-23T00:33:57.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.665+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ba410a3b0 con 0x7f3ba40f5410 2024-02-23T00:33:57.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.666+0000 7f3b8bfff700 1 -- 172.21.15.118:0/2924340120 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b9401db70 con 0x7f3ba40f5410 2024-02-23T00:33:57.667 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:33:57.668 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.667+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 msgr2=0x7f3b8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.667+0000 7f3baa124700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 0x7f3b8c0323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3b980096a0 tx=0x7f3b98006b40).stop 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 msgr2=0x7f3ba41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f3b94005010 tx=0x7f3b9400b040).stop 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 shutdown_connections 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f3b8c02ff10 0x7f3b8c0323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 --2- 172.21.15.118:0/2924340120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ba40f5410 0x7f3ba41094e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:57.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 >> 172.21.15.118:0/2924340120 conn(0x7f3ba40f0b70 msgr2=0x7f3ba40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:57.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 shutdown_connections 2024-02-23T00:33:57.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:33:57.668+0000 7f3baa124700 1 -- 172.21.15.118:0/2924340120 wait complete. 2024-02-23T00:33:57.670 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:33:58.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:56.345020+0000 mgr.smithi090.pxxmlz (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:33:58.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:33:58 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:33:57.666896+0000 mon.smithi090 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.118:0/2924340120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:33:59.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:33:59.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:00.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:33:58.345276+0000 mgr.smithi090.pxxmlz (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.618+0000 7f894955d700 1 -- 172.21.15.118:0/2288357089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 msgr2=0x7f89440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 --2- 172.21.15.118:0/2288357089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f89440f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f8934009fb0 tx=0x7f8934009b30).stop 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- 172.21.15.118:0/2288357089 shutdown_connections 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 --2- 172.21.15.118:0/2288357089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f89440f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- 172.21.15.118:0/2288357089 >> 172.21.15.118:0/2288357089 conn(0x7f89440f09a0 msgr2=0x7f89440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- 172.21.15.118:0/2288357089 shutdown_connections 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- 172.21.15.118:0/2288357089 wait complete. 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 Processor -- start 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- start start 2024-02-23T00:34:01.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.619+0000 7f894955d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441070e0 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8943fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8943fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:41366/0 (socket says 172.21.15.118:41366) 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8943fff700 1 -- 172.21.15.118:0/2715100860 learned_addr learned my addr 172.21.15.118:0/2715100860 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8943fff700 1 -- 172.21.15.118:0/2715100860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934005040 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8943fff700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f89340089f0 tx=0x7f893400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8934009320 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8944060220 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8934013380 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8934014e20 con 0x7f89440f51a0 2024-02-23T00:34:01.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.620+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894410bbe0 con 0x7f89440f51a0 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.621+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8934014420 con 0x7f89440f51a0 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.621+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8944045ad0 con 0x7f89440f51a0 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.621+0000 7f8940ff9700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.621+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8934040f50 con 0x7f89440f51a0 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.622+0000 7f89437fe700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.622+0000 7f89437fe700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 0x7f892c0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f89380096a0 tx=0x7f8938006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.626+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8934014790 con 0x7f89440f51a0 2024-02-23T00:34:01.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.833+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f894410c3d0 con 0x7f89440f51a0 2024-02-23T00:34:01.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.834+0000 7f8940ff9700 1 -- 172.21.15.118:0/2715100860 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f893401d7f0 con 0x7f89440f51a0 2024-02-23T00:34:01.835 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:01.835 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 msgr2=0x7f892c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 0x7f892c0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f89380096a0 tx=0x7f8938006b40).stop 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 msgr2=0x7f894410b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f89340089f0 tx=0x7f893400b040).stop 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 shutdown_connections 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f892c02ff10 0x7f892c0323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 --2- 172.21.15.118:0/2715100860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89440f51a0 0x7f894410b6f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 >> 172.21.15.118:0/2715100860 conn(0x7f89440f09a0 msgr2=0x7f89440f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 shutdown_connections 2024-02-23T00:34:01.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:01.835+0000 7f894955d700 1 -- 172.21.15.118:0/2715100860 wait complete. 2024-02-23T00:34:01.838 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:02.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:00.345624+0000 mgr.smithi090.pxxmlz (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:02.524 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:02 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:01.834604+0000 mon.smithi090 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.118:0/2715100860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:03.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:03.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:04.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:02.346016+0000 mgr.smithi090.pxxmlz (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/2698427904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 msgr2=0x7fc6c00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/2698427904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c00f7d00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc6b4009fb0 tx=0x7fc6b4009b30).stop 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/2698427904 shutdown_connections 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/2698427904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c00f7d00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/2698427904 >> 172.21.15.118:0/2698427904 conn(0x7fc6c00f0120 msgr2=0x7fc6c00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/2698427904 shutdown_connections 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.647+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/2698427904 wait complete. 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.648+0000 7fc6c4d9d700 1 Processor -- start 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.648+0000 7fc6c4d9d700 1 -- start start 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6c4d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:05.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6c4d9d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0106ad0 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6bf7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6bf7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48164/0 (socket says 172.21.15.118:48164) 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6bf7fe700 1 -- 172.21.15.118:0/3339706933 learned_addr learned my addr 172.21.15.118:0/3339706933 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6bf7fe700 1 -- 172.21.15.118:0/3339706933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4005040 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6bf7fe700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc6b40089f0 tx=0x7fc6b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6b4009320 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c0106cd0 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6b4013380 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6b4014e20 con 0x7fc6c00f78e0 2024-02-23T00:34:05.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.649+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c0107160 con 0x7fc6c00f78e0 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.650+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6b40263a0 con 0x7fc6c00f78e0 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.650+0000 7fc6a7fff700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 0x7fc6a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.650+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c0045ad0 con 0x7fc6c00f78e0 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.650+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6b4041270 con 0x7fc6c00f78e0 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.650+0000 7fc6beffd700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 0x7fc6a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:05.653 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.651+0000 7fc6beffd700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 0x7fc6a80323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc6b00096a0 tx=0x7fc6b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:05.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.655+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6b401cac0 con 0x7fc6c00f78e0 2024-02-23T00:34:05.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.862+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6c0107f40 con 0x7fc6c00f78e0 2024-02-23T00:34:05.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.863+0000 7fc6a7fff700 1 -- 172.21.15.118:0/3339706933 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6b402c090 con 0x7fc6c00f78e0 2024-02-23T00:34:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:05.865 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:05.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 msgr2=0x7fc6a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:05.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 0x7fc6a80323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc6b00096a0 tx=0x7fc6b0006b40).stop 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 msgr2=0x7fc6c01065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc6b40089f0 tx=0x7fc6b400b040).stop 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 shutdown_connections 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc6a802ff10 0x7fc6a80323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 --2- 172.21.15.118:0/3339706933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6c00f78e0 0x7fc6c01065e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 >> 172.21.15.118:0/3339706933 conn(0x7fc6c00f0120 msgr2=0x7fc6c00f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 shutdown_connections 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:05.865+0000 7fc6c4d9d700 1 -- 172.21.15.118:0/3339706933 wait complete. 2024-02-23T00:34:05.867 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:06.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:04.346356+0000 mgr.smithi090.pxxmlz (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:06.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:06 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:05.864066+0000 mon.smithi090 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.118:0/3339706933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:07.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:07.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:08.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:06.346652+0000 mgr.smithi090.pxxmlz (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:09.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.894+0000 7f4be4d43700 1 -- 172.21.15.118:0/909383905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 msgr2=0x7f4be00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:09.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.894+0000 7f4be4d43700 1 --2- 172.21.15.118:0/909383905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be00f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4bc8009fb0 tx=0x7f4bc8009b30).stop 2024-02-23T00:34:09.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- 172.21.15.118:0/909383905 shutdown_connections 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 --2- 172.21.15.118:0/909383905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be00f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- 172.21.15.118:0/909383905 >> 172.21.15.118:0/909383905 conn(0x7f4be00f0b70 msgr2=0x7f4be00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- 172.21.15.118:0/909383905 shutdown_connections 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- 172.21.15.118:0/909383905 wait complete. 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 Processor -- start 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- start start 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4be4d43700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be0060240 con 0x7f4be00f5410 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4bde59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:09.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4bde59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48182/0 (socket says 172.21.15.118:48182) 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.895+0000 7f4bde59c700 1 -- 172.21.15.118:0/2201797482 learned_addr learned my addr 172.21.15.118:0/2201797482 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4bde59c700 1 -- 172.21.15.118:0/2201797482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc8005040 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4bde59c700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4bc8005010 tx=0x7f4bc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc8009320 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bc8013380 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be0060440 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bc8014e50 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.896+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be0109b90 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bc8014420 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4bd6ffd700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 0x7f4bcc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be0045ad0 con 0x7f4be00f5410 2024-02-23T00:34:09.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bc8040f40 con 0x7f4be00f5410 2024-02-23T00:34:09.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4bddd9b700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 0x7f4bcc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:09.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.897+0000 7f4bddd9b700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 0x7f4bcc0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4bd00096a0 tx=0x7f4bd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:09.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:09.901+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bc801db70 con 0x7f4be00f5410 2024-02-23T00:34:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.110+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4be010a3b0 con 0x7f4be00f5410 2024-02-23T00:34:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.111+0000 7f4bd6ffd700 1 -- 172.21.15.118:0/2201797482 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bc801db70 con 0x7f4be00f5410 2024-02-23T00:34:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:10.113 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 msgr2=0x7f4bcc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 0x7f4bcc0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4bd00096a0 tx=0x7f4bd0006b40).stop 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 msgr2=0x7f4be01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f4bc8005010 tx=0x7f4bc800b040).stop 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 shutdown_connections 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4bcc02ff10 0x7f4bcc0323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 --2- 172.21.15.118:0/2201797482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4be00f5410 0x7f4be01094e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:10.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 >> 172.21.15.118:0/2201797482 conn(0x7f4be00f0b70 msgr2=0x7f4be00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:10.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 shutdown_connections 2024-02-23T00:34:10.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:10.112+0000 7f4be4d43700 1 -- 172.21.15.118:0/2201797482 wait complete. 2024-02-23T00:34:10.117 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:10.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:10 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:08.347044+0000 mgr.smithi090.pxxmlz (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:11.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:10.111715+0000 mon.smithi090 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.118:0/2201797482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:11.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:11.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:12.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:12 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:10.347310+0000 mgr.smithi090.pxxmlz (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 -- 172.21.15.118:0/149354230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 msgr2=0x7ff2b40f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/149354230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b40f6850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7ff2a4009fb0 tx=0x7ff2a4009b30).stop 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 -- 172.21.15.118:0/149354230 shutdown_connections 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/149354230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b40f6850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 -- 172.21.15.118:0/149354230 >> 172.21.15.118:0/149354230 conn(0x7ff2b40f1b90 msgr2=0x7ff2b40f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 -- 172.21.15.118:0/149354230 shutdown_connections 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.975+0000 7ff2ba30e700 1 -- 172.21.15.118:0/149354230 wait complete. 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2ba30e700 1 Processor -- start 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2ba30e700 1 -- start start 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2ba30e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2ba30e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b4106f70 con 0x7ff2b40f6430 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2b3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2b3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:48198/0 (socket says 172.21.15.118:48198) 2024-02-23T00:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.976+0000 7ff2b3fff700 1 -- 172.21.15.118:0/1125899642 learned_addr learned my addr 172.21.15.118:0/1125899642 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2b3fff700 1 -- 172.21.15.118:0/1125899642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4005040 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2b3fff700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7ff2a40089f0 tx=0x7ff2a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a4009320 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b4107170 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2a4013380 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a401ed80 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.977+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b4107600 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.978+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2a4014420 con 0x7ff2b40f6430 2024-02-23T00:34:13.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.978+0000 7ff2b0ff9700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:13.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.978+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2a4040f80 con 0x7ff2b40f6430 2024-02-23T00:34:13.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.978+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b4045ad0 con 0x7ff2b40f6430 2024-02-23T00:34:13.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.978+0000 7ff2b37fe700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:13.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.979+0000 7ff2b37fe700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 0x7ff29c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ff2a80096a0 tx=0x7ff2a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:13.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:13.983+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2a401c910 con 0x7ff2b40f6430 2024-02-23T00:34:14.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.192+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2b405ea50 con 0x7ff2b40f6430 2024-02-23T00:34:14.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.193+0000 7ff2b0ff9700 1 -- 172.21.15.118:0/1125899642 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2a401c320 con 0x7ff2b40f6430 2024-02-23T00:34:14.194 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:14.194 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 msgr2=0x7ff29c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 0x7ff29c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ff2a80096a0 tx=0x7ff2a8006b40).stop 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 msgr2=0x7ff2b4106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7ff2a40089f0 tx=0x7ff2a400b040).stop 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 shutdown_connections 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff29c02ff60 0x7ff29c032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:14.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 --2- 172.21.15.118:0/1125899642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2b40f6430 0x7ff2b4106a80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:14.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 >> 172.21.15.118:0/1125899642 conn(0x7ff2b40f1b90 msgr2=0x7ff2b40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:14.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.194+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 shutdown_connections 2024-02-23T00:34:14.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:14.195+0000 7ff2ba30e700 1 -- 172.21.15.118:0/1125899642 wait complete. 2024-02-23T00:34:14.197 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:14.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:14 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:12.347632+0000 mgr.smithi090.pxxmlz (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:15.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:14.193637+0000 mon.smithi090 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.118:0/1125899642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:16.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:16.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:16.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:14.348016+0000 mgr.smithi090.pxxmlz (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 -- 172.21.15.118:0/4107944407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 msgr2=0x7faac40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 --2- 172.21.15.118:0/4107944407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac40f66c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7faab4009fb0 tx=0x7faab4009b30).stop 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 -- 172.21.15.118:0/4107944407 shutdown_connections 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 --2- 172.21.15.118:0/4107944407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac40f66c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 -- 172.21.15.118:0/4107944407 >> 172.21.15.118:0/4107944407 conn(0x7faac40f1aa0 msgr2=0x7faac40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 -- 172.21.15.118:0/4107944407 shutdown_connections 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 -- 172.21.15.118:0/4107944407 wait complete. 2024-02-23T00:34:18.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.202+0000 7faac8e92700 1 Processor -- start 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac8e92700 1 -- start start 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac8e92700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac8e92700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac4106ac0 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac37fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53188/0 (socket says 172.21.15.118:53188) 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac37fe700 1 -- 172.21.15.118:0/1059191333 learned_addr learned my addr 172.21.15.118:0/1059191333 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.203+0000 7faac37fe700 1 -- 172.21.15.118:0/1059191333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4005040 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faac37fe700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7faab40089f0 tx=0x7faab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab4009320 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac4106cc0 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faab4013380 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab4014e20 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.204+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac4107150 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faab40095f0 con 0x7faac40f62a0 2024-02-23T00:34:18.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac4045ad0 con 0x7faac40f62a0 2024-02-23T00:34:18.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faaabfff700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 0x7faaac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faab4040ea0 con 0x7faac40f62a0 2024-02-23T00:34:18.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faac2ffd700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 0x7faaac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.205+0000 7faac2ffd700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 0x7faaac0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7faab80096a0 tx=0x7faab8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.209+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faab4014420 con 0x7faac40f62a0 2024-02-23T00:34:18.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.418+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faac4108000 con 0x7faac40f62a0 2024-02-23T00:34:18.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.418+0000 7faaabfff700 1 -- 172.21.15.118:0/1059191333 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faab4014420 con 0x7faac40f62a0 2024-02-23T00:34:18.420 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:18.420 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:18.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 msgr2=0x7faaac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 0x7faaac0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7faab80096a0 tx=0x7faab8006b40).stop 2024-02-23T00:34:18.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 msgr2=0x7faac41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7faab40089f0 tx=0x7faab400b040).stop 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 shutdown_connections 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7faaac02ff10 0x7faaac0323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 --2- 172.21.15.118:0/1059191333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac40f62a0 0x7faac41065d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 >> 172.21.15.118:0/1059191333 conn(0x7faac40f1aa0 msgr2=0x7faac40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 shutdown_connections 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:18.420+0000 7faac8e92700 1 -- 172.21.15.118:0/1059191333 wait complete. 2024-02-23T00:34:18.422 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:18.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:18 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:16.348341+0000 mgr.smithi090.pxxmlz (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:19.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:18.419304+0000 mon.smithi090 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.118:0/1059191333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:20.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:20.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:20.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:18.348609+0000 mgr.smithi090.pxxmlz (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:22.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 -- 172.21.15.118:0/2181731495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 msgr2=0x7f8bd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:22.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 --2- 172.21.15.118:0/2181731495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd40f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8bbc009fb0 tx=0x7f8bbc009b30).stop 2024-02-23T00:34:22.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 -- 172.21.15.118:0/2181731495 shutdown_connections 2024-02-23T00:34:22.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 --2- 172.21.15.118:0/2181731495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd40f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 -- 172.21.15.118:0/2181731495 >> 172.21.15.118:0/2181731495 conn(0x7f8bd40f1c70 msgr2=0x7f8bd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 -- 172.21.15.118:0/2181731495 shutdown_connections 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.242+0000 7f8bd9121700 1 -- 172.21.15.118:0/2181731495 wait complete. 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.243+0000 7f8bd9121700 1 Processor -- start 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.243+0000 7f8bd9121700 1 -- start start 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.243+0000 7f8bd9121700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.243+0000 7f8bd9121700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd4106cd0 con 0x7f8bd40f6510 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.243+0000 7f8bd2d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd2d9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53200/0 (socket says 172.21.15.118:53200) 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd2d9d700 1 -- 172.21.15.118:0/4151707101 learned_addr learned my addr 172.21.15.118:0/4151707101 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd2d9d700 1 -- 172.21.15.118:0/4151707101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc005040 con 0x7f8bd40f6510 2024-02-23T00:34:22.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd2d9d700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8bbc005010 tx=0x7f8bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bbc009320 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd4106ed0 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bbc013380 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bbc014e50 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.244+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd4107360 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.245+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bbc0095f0 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.245+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd4045ad0 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.245+0000 7f8bcb7fe700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 0x7f8bc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.245+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bbc040e90 con 0x7f8bd40f6510 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.245+0000 7f8bd259c700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 0x7f8bc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.246+0000 7f8bd259c700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 0x7f8bc00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8bc40096a0 tx=0x7f8bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:22.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.250+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bbc014420 con 0x7f8bd40f6510 2024-02-23T00:34:22.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.454+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bd4108210 con 0x7f8bd40f6510 2024-02-23T00:34:22.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.455+0000 7f8bcb7fe700 1 -- 172.21.15.118:0/4151707101 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8bbc010230 con 0x7f8bd40f6510 2024-02-23T00:34:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:22.457 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:22.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 msgr2=0x7f8bc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:22.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 0x7f8bc00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8bc40096a0 tx=0x7f8bc4006b40).stop 2024-02-23T00:34:22.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 msgr2=0x7f8bd41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8bbc005010 tx=0x7f8bbc00b040).stop 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 shutdown_connections 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8bc002ff10 0x7f8bc00323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 --2- 172.21.15.118:0/4151707101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd40f6510 0x7f8bd41067e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 >> 172.21.15.118:0/4151707101 conn(0x7f8bd40f1c70 msgr2=0x7f8bd40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 shutdown_connections 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:22.457+0000 7f8bd9121700 1 -- 172.21.15.118:0/4151707101 wait complete. 2024-02-23T00:34:22.459 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:22.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:22 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:20.348855+0000 mgr.smithi090.pxxmlz (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:23.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:22.456165+0000 mon.smithi090 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.118:0/4151707101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:24.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:24.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:24.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:22.349191+0000 mgr.smithi090.pxxmlz (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.489+0000 7ff0b3e29700 1 -- 172.21.15.118:0/2189969928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 msgr2=0x7ff0ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.489+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/2189969928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac0f90f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff0a8009fb0 tx=0x7ff0a8009b30).stop 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- 172.21.15.118:0/2189969928 shutdown_connections 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/2189969928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac0f90f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- 172.21.15.118:0/2189969928 >> 172.21.15.118:0/2189969928 conn(0x7ff0ac0f0b70 msgr2=0x7ff0ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- 172.21.15.118:0/2189969928 shutdown_connections 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- 172.21.15.118:0/2189969928 wait complete. 2024-02-23T00:34:26.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 Processor -- start 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- start start 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.490+0000 7ff0b3e29700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac106c70 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b1bc5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b1bc5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55826/0 (socket says 172.21.15.118:55826) 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b1bc5700 1 -- 172.21.15.118:0/1524270769 learned_addr learned my addr 172.21.15.118:0/1524270769 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b1bc5700 1 -- 172.21.15.118:0/1524270769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a8005040 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b1bc5700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff0a80089f0 tx=0x7ff0a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0a8009320 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ac106e70 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0a8013380 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0a8014e20 con 0x7ff0ac060280 2024-02-23T00:34:26.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.491+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ac107300 con 0x7ff0ac060280 2024-02-23T00:34:26.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.492+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac045ad0 con 0x7ff0ac060280 2024-02-23T00:34:26.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.492+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0a80095f0 con 0x7ff0ac060280 2024-02-23T00:34:26.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.493+0000 7ff0a27fc700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 0x7ff0980367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:26.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.493+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0a8040e90 con 0x7ff0ac060280 2024-02-23T00:34:26.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.493+0000 7ff0b13c4700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 0x7ff0980367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:26.496 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.493+0000 7ff0b13c4700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 0x7ff0980367e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff09c0096a0 tx=0x7ff09c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:26.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.497+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0a8014420 con 0x7ff0ac060280 2024-02-23T00:34:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:24.349443+0000 mgr.smithi090.pxxmlz (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:25.254532+0000 mon.smithi090 (mon.0) 308 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:34:26.523 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:25.260138+0000 mon.smithi090 (mon.0) 309 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:34:26.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.703+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0ac1081b0 con 0x7ff0ac060280 2024-02-23T00:34:26.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.703+0000 7ff0a27fc700 1 -- 172.21.15.118:0/1524270769 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0a801d980 con 0x7ff0ac060280 2024-02-23T00:34:26.705 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:26.705 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 msgr2=0x7ff0980367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 0x7ff0980367e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7ff09c0096a0 tx=0x7ff09c006b40).stop 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 msgr2=0x7ff0ac106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7ff0a80089f0 tx=0x7ff0a800b040).stop 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 shutdown_connections 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7ff098034320 0x7ff0980367e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 --2- 172.21.15.118:0/1524270769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0ac060280 0x7ff0ac106780 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 >> 172.21.15.118:0/1524270769 conn(0x7ff0ac0f0b70 msgr2=0x7ff0ac0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 shutdown_connections 2024-02-23T00:34:26.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:26.705+0000 7ff0b3e29700 1 -- 172.21.15.118:0/1524270769 wait complete. 2024-02-23T00:34:26.708 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:27.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:26.704357+0000 mon.smithi090 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.118:0/1524270769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:28.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:28.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:28.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:26.349683+0000 mgr.smithi090.pxxmlz (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:30.522 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:30 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:28.349985+0000 mgr.smithi090.pxxmlz (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 -- 172.21.15.118:0/447717620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 msgr2=0x7f50540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 --2- 172.21.15.118:0/447717620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f50540f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f5048009fb0 tx=0x7f5048009b30).stop 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 -- 172.21.15.118:0/447717620 shutdown_connections 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 --2- 172.21.15.118:0/447717620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f50540f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 -- 172.21.15.118:0/447717620 >> 172.21.15.118:0/447717620 conn(0x7f50540f1c70 msgr2=0x7f50540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:30.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 -- 172.21.15.118:0/447717620 shutdown_connections 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.656+0000 7f505bd41700 1 -- 172.21.15.118:0/447717620 wait complete. 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f505bd41700 1 Processor -- start 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f505bd41700 1 -- start start 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f505bd41700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f505bd41700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5054069e20 con 0x7f50540f6510 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f5059add700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f5059add700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55848/0 (socket says 172.21.15.118:55848) 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.657+0000 7f5059add700 1 -- 172.21.15.118:0/2339490254 learned_addr learned my addr 172.21.15.118:0/2339490254 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f5059add700 1 -- 172.21.15.118:0/2339490254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048005040 con 0x7f50540f6510 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f5059add700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f5048005010 tx=0x7f504800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5048009320 con 0x7f50540f6510 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5054069fc0 con 0x7f50540f6510 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5048013380 con 0x7f50540f6510 2024-02-23T00:34:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5048014e50 con 0x7f50540f6510 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.658+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505406a450 con 0x7f50540f6510 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.659+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5048014420 con 0x7f50540f6510 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.659+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5054059100 con 0x7f50540f6510 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.659+0000 7f50467fc700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 0x7f50400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.659+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5048040f40 con 0x7f50540f6510 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.659+0000 7f50592dc700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 0x7f50400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.660+0000 7f50592dc700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 0x7f50400323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f50500096a0 tx=0x7f5050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:30.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.664+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50480095f0 con 0x7f50540f6510 2024-02-23T00:34:30.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.872+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50540fc0a0 con 0x7f50540f6510 2024-02-23T00:34:30.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.873+0000 7f50467fc700 1 -- 172.21.15.118:0/2339490254 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f504801db70 con 0x7f50540f6510 2024-02-23T00:34:30.874 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:30.875 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 msgr2=0x7f50400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 0x7f50400323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f50500096a0 tx=0x7f5050006b40).stop 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 msgr2=0x7f505406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f5048005010 tx=0x7f504800b040).stop 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 shutdown_connections 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f504002ff10 0x7f50400323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 --2- 172.21.15.118:0/2339490254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50540f6510 0x7f505406b9a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 >> 172.21.15.118:0/2339490254 conn(0x7f50540f1c70 msgr2=0x7f50540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 shutdown_connections 2024-02-23T00:34:30.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:30.874+0000 7f505bd41700 1 -- 172.21.15.118:0/2339490254 wait complete. 2024-02-23T00:34:30.877 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:30.350253+0000 mgr.smithi090.pxxmlz (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:31.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:30.873790+0000 mon.smithi090 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.118:0/2339490254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:32.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:32.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:33.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:32.350498+0000 mgr.smithi090.pxxmlz (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:34.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 -- 172.21.15.118:0/4077221943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 msgr2=0x7f73980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:34.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 --2- 172.21.15.118:0/4077221943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f73980f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f7388009fb0 tx=0x7f7388009b30).stop 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 -- 172.21.15.118:0/4077221943 shutdown_connections 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 --2- 172.21.15.118:0/4077221943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f73980f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 -- 172.21.15.118:0/4077221943 >> 172.21.15.118:0/4077221943 conn(0x7f73980f1c70 msgr2=0x7f73980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 -- 172.21.15.118:0/4077221943 shutdown_connections 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.782+0000 7f739f42f700 1 -- 172.21.15.118:0/4077221943 wait complete. 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739f42f700 1 Processor -- start 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739f42f700 1 -- start start 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739f42f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739f42f700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398069e20 con 0x7f73980f6510 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739d1cb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739d1cb700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33356/0 (socket says 172.21.15.118:33356) 2024-02-23T00:34:34.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739d1cb700 1 -- 172.21.15.118:0/1314125024 learned_addr learned my addr 172.21.15.118:0/1314125024 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739d1cb700 1 -- 172.21.15.118:0/1314125024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388005040 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.783+0000 7f739d1cb700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f7388005010 tx=0x7f738800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.784+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7388009320 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.784+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7398069fc0 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.784+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7388013380 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.784+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7388014e50 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.784+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739806a450 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.785+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73880095f0 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.785+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7398059100 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.785+0000 7f738dffb700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 0x7f73840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.785+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7388040e90 con 0x7f73980f6510 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.785+0000 7f739c9ca700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 0x7f73840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:34.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.786+0000 7f739c9ca700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 0x7f73840323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f73940096a0 tx=0x7f7394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:34.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.790+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7388014420 con 0x7f73980f6510 2024-02-23T00:34:34.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.994+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73980fc0a0 con 0x7f73980f6510 2024-02-23T00:34:34.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.995+0000 7f738dffb700 1 -- 172.21.15.118:0/1314125024 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7388010230 con 0x7f73980f6510 2024-02-23T00:34:34.998 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:34.998 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:34.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 msgr2=0x7f73840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:34.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 0x7f73840323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f73940096a0 tx=0x7f7394006b40).stop 2024-02-23T00:34:34.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 msgr2=0x7f739806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f7388005010 tx=0x7f738800b040).stop 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 shutdown_connections 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f738402ff10 0x7f73840323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 --2- 172.21.15.118:0/1314125024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73980f6510 0x7f739806b9a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 >> 172.21.15.118:0/1314125024 conn(0x7f73980f1c70 msgr2=0x7f73980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 shutdown_connections 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:34.997+0000 7f739f42f700 1 -- 172.21.15.118:0/1314125024 wait complete. 2024-02-23T00:34:35.000 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:36.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:34.350862+0000 mgr.smithi090.pxxmlz (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:36.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:34.996914+0000 mon.smithi090 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.118:0/1314125024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:36.910 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:36.910 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:38.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:36.351200+0000 mgr.smithi090.pxxmlz (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:39.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.137+0000 7f8b992a0700 1 -- 172.21.15.118:0/302293336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 msgr2=0x7f8b940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.137+0000 7f8b992a0700 1 --2- 172.21.15.118:0/302293336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b940f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f8b88009fb0 tx=0x7f8b88009b30).stop 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.137+0000 7f8b992a0700 1 -- 172.21.15.118:0/302293336 shutdown_connections 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.137+0000 7f8b992a0700 1 --2- 172.21.15.118:0/302293336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b940f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.137+0000 7f8b992a0700 1 -- 172.21.15.118:0/302293336 >> 172.21.15.118:0/302293336 conn(0x7f8b940f0b70 msgr2=0x7f8b940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 -- 172.21.15.118:0/302293336 shutdown_connections 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 -- 172.21.15.118:0/302293336 wait complete. 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 Processor -- start 2024-02-23T00:34:39.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 -- start start 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.138+0000 7f8b992a0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94106d10 con 0x7f8b940f5410 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b92ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b92ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33382/0 (socket says 172.21.15.118:33382) 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b92ffd700 1 -- 172.21.15.118:0/1552248964 learned_addr learned my addr 172.21.15.118:0/1552248964 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:39.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b92ffd700 1 -- 172.21.15.118:0/1552248964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88005040 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b92ffd700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f8b88005010 tx=0x7f8b8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b88009320 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b94106f10 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b88013380 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b88014e50 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.139+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b941073a0 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.140+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b88014420 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.140+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b94045ad0 con 0x7f8b940f5410 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.140+0000 7f8b7b7fe700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 0x7f8b7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.140+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b88040f70 con 0x7f8b940f5410 2024-02-23T00:34:39.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.141+0000 7f8b927fc700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 0x7f8b7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.141+0000 7f8b927fc700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 0x7f8b7c0323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8b840096a0 tx=0x7f8b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:39.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.145+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b880095f0 con 0x7f8b940f5410 2024-02-23T00:34:39.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.350+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b94108250 con 0x7f8b940f5410 2024-02-23T00:34:39.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.351+0000 7f8b7b7fe700 1 -- 172.21.15.118:0/1552248964 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b8801db70 con 0x7f8b940f5410 2024-02-23T00:34:39.353 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:39.353 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 msgr2=0x7f8b7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 0x7f8b7c0323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8b840096a0 tx=0x7f8b84006b40).stop 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 msgr2=0x7f8b94106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f8b88005010 tx=0x7f8b8800b040).stop 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 shutdown_connections 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8b7c02ff10 0x7f8b7c0323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 --2- 172.21.15.118:0/1552248964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b940f5410 0x7f8b94106820 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 >> 172.21.15.118:0/1552248964 conn(0x7f8b940f0b70 msgr2=0x7f8b940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 shutdown_connections 2024-02-23T00:34:39.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:39.352+0000 7f8b992a0700 1 -- 172.21.15.118:0/1552248964 wait complete. 2024-02-23T00:34:39.356 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:40.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:38.351466+0000 mgr.smithi090.pxxmlz (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:40.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:39.352498+0000 mon.smithi090 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.118:0/1552248964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:41.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:41.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:42.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:40.351715+0000 mgr.smithi090.pxxmlz (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:43.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 -- 172.21.15.118:0/851438883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 msgr2=0x7fcbc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/851438883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc80f66c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fcbbc009fb0 tx=0x7fcbbc009b30).stop 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 -- 172.21.15.118:0/851438883 shutdown_connections 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/851438883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc80f66c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 -- 172.21.15.118:0/851438883 >> 172.21.15.118:0/851438883 conn(0x7fcbc80f1aa0 msgr2=0x7fcbc80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 -- 172.21.15.118:0/851438883 shutdown_connections 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.336+0000 7fcbc7fff700 1 -- 172.21.15.118:0/851438883 wait complete. 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc7fff700 1 Processor -- start 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc7fff700 1 -- start start 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc7fff700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc8069e20 con 0x7fcbc80f62a0 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc6ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:33396/0 (socket says 172.21.15.118:33396) 2024-02-23T00:34:43.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc6ffd700 1 -- 172.21.15.118:0/4166375311 learned_addr learned my addr 172.21.15.118:0/4166375311 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.337+0000 7fcbc6ffd700 1 -- 172.21.15.118:0/4166375311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc005040 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbc6ffd700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fcbbc0089f0 tx=0x7fcbbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbbc009320 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc8069fc0 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc806a450 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbbc013380 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.338+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbbc014e20 con 0x7fcbc80f62a0 2024-02-23T00:34:43.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbbc0263a0 con 0x7fcbc80f62a0 2024-02-23T00:34:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbaf7fe700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 0x7fcbb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbbc040fe0 con 0x7fcbc80f62a0 2024-02-23T00:34:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc8059100 con 0x7fcbc80f62a0 2024-02-23T00:34:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbc67fc700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 0x7fcbb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.339+0000 7fcbc67fc700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 0x7fcbb00323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcbb80096a0 tx=0x7fcbb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.344+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbbc012920 con 0x7fcbc80f62a0 2024-02-23T00:34:43.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.548+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbc80fbe30 con 0x7fcbc80f62a0 2024-02-23T00:34:43.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.549+0000 7fcbaf7fe700 1 -- 172.21.15.118:0/4166375311 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbbc012330 con 0x7fcbc80f62a0 2024-02-23T00:34:43.551 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:43.551 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:43.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 msgr2=0x7fcbb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 0x7fcbb00323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcbb80096a0 tx=0x7fcbb8006b40).stop 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 msgr2=0x7fcbc806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fcbbc0089f0 tx=0x7fcbbc00b040).stop 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 shutdown_connections 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcbb002ff10 0x7fcbb00323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 --2- 172.21.15.118:0/4166375311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbc80f62a0 0x7fcbc806b9a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 >> 172.21.15.118:0/4166375311 conn(0x7fcbc80f1aa0 msgr2=0x7fcbc80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 shutdown_connections 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:43.551+0000 7fcbc7fff700 1 -- 172.21.15.118:0/4166375311 wait complete. 2024-02-23T00:34:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:44.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:42.352010+0000 mgr.smithi090.pxxmlz (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:45.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:43.550183+0000 mon.smithi090 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.118:0/4166375311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:45.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:45.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:46.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:44.352259+0000 mgr.smithi090.pxxmlz (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:47.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 -- 172.21.15.118:0/3458234595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 msgr2=0x7fd2b40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/3458234595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b40f9150 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fd2a8009fb0 tx=0x7fd2a8009b30).stop 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 -- 172.21.15.118:0/3458234595 shutdown_connections 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/3458234595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b40f9150 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 -- 172.21.15.118:0/3458234595 >> 172.21.15.118:0/3458234595 conn(0x7fd2b40f0b70 msgr2=0x7fd2b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 -- 172.21.15.118:0/3458234595 shutdown_connections 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.136+0000 7fd2b97d2700 1 -- 172.21.15.118:0/3458234595 wait complete. 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b97d2700 1 Processor -- start 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b97d2700 1 -- start start 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b97d2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b97d2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4106cb0 con 0x7fd2b4060220 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b2ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:46722/0 (socket says 172.21.15.118:46722) 2024-02-23T00:34:47.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.137+0000 7fd2b2ffd700 1 -- 172.21.15.118:0/4086747639 learned_addr learned my addr 172.21.15.118:0/4086747639 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd2b2ffd700 1 -- 172.21.15.118:0/4086747639 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a8005040 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd2b2ffd700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd2a80089f0 tx=0x7fd2a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2a8009320 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b4106eb0 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2a8013380 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2a8014e20 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.138+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b4107340 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.139+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2a80095f0 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.139+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b4045ad0 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.139+0000 7fd29b7fe700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 0x7fd29c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.139+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2a8040f50 con 0x7fd2b4060220 2024-02-23T00:34:47.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.139+0000 7fd2b27fc700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 0x7fd29c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:47.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.140+0000 7fd2b27fc700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 0x7fd29c0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd2a40096a0 tx=0x7fd2a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:47.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.144+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2a8014420 con 0x7fd2b4060220 2024-02-23T00:34:47.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.349+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2b41081f0 con 0x7fd2b4060220 2024-02-23T00:34:47.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.350+0000 7fd29b7fe700 1 -- 172.21.15.118:0/4086747639 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd2a801d980 con 0x7fd2b4060220 2024-02-23T00:34:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:47.352 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:47.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 msgr2=0x7fd29c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 0x7fd29c0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd2a40096a0 tx=0x7fd2a4006b40).stop 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 msgr2=0x7fd2b41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd2a80089f0 tx=0x7fd2a800b040).stop 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 shutdown_connections 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fd29c02ff10 0x7fd29c0323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 --2- 172.21.15.118:0/4086747639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2b4060220 0x7fd2b41067c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 >> 172.21.15.118:0/4086747639 conn(0x7fd2b40f0b70 msgr2=0x7fd2b40f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 shutdown_connections 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:47.352+0000 7fd2b97d2700 1 -- 172.21.15.118:0/4086747639 wait complete. 2024-02-23T00:34:47.354 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:48.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:46.352540+0000 mgr.smithi090.pxxmlz (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:48.095 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:47.350996+0000 mon.smithi090 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.118:0/4086747639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:49.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:49.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:50.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:48.352815+0000 mgr.smithi090.pxxmlz (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:51.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 -- 172.21.15.118:0/3448027256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 msgr2=0x7f62540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:51.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 --2- 172.21.15.118:0/3448027256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62540f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f6248009fb0 tx=0x7f6248009b30).stop 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 -- 172.21.15.118:0/3448027256 shutdown_connections 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 --2- 172.21.15.118:0/3448027256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62540f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 -- 172.21.15.118:0/3448027256 >> 172.21.15.118:0/3448027256 conn(0x7f62540f0b70 msgr2=0x7f62540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 -- 172.21.15.118:0/3448027256 shutdown_connections 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.275+0000 7f625adc5700 1 -- 172.21.15.118:0/3448027256 wait complete. 2024-02-23T00:34:51.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.276+0000 7f625adc5700 1 Processor -- start 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.276+0000 7f625adc5700 1 -- start start 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.276+0000 7f625adc5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.276+0000 7f625adc5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6254106cd0 con 0x7f62540f5410 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.276+0000 7f6258b61700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f6258b61700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:46736/0 (socket says 172.21.15.118:46736) 2024-02-23T00:34:51.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f6258b61700 1 -- 172.21.15.118:0/2808426579 learned_addr learned my addr 172.21.15.118:0/2808426579 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f6258b61700 1 -- 172.21.15.118:0/2808426579 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6248005040 con 0x7f62540f5410 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f6258b61700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6248005010 tx=0x7f624800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6248009320 con 0x7f62540f5410 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6248013380 con 0x7f62540f5410 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6254106ed0 con 0x7f62540f5410 2024-02-23T00:34:51.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6248014e50 con 0x7f62540f5410 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.277+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6254107360 con 0x7f62540f5410 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.278+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62480095f0 con 0x7f62540f5410 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.278+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6254045ad0 con 0x7f62540f5410 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.278+0000 7f62517fa700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 0x7f62440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.278+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6248040e90 con 0x7f62540f5410 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.278+0000 7f6253fff700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 0x7f62440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:51.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.279+0000 7f6253fff700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 0x7f62440323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f62400096a0 tx=0x7f6240006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:51.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.283+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6248014420 con 0x7f62540f5410 2024-02-23T00:34:51.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.489+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62540fafa0 con 0x7f62540f5410 2024-02-23T00:34:51.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.490+0000 7f62517fa700 1 -- 172.21.15.118:0/2808426579 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6248010230 con 0x7f62540f5410 2024-02-23T00:34:51.492 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:51.492 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 msgr2=0x7f62440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 0x7f62440323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f62400096a0 tx=0x7f6240006b40).stop 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 msgr2=0x7f62541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6248005010 tx=0x7f624800b040).stop 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 shutdown_connections 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f624402ff10 0x7f62440323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 --2- 172.21.15.118:0/2808426579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62540f5410 0x7f62541067e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 >> 172.21.15.118:0/2808426579 conn(0x7f62540f0b70 msgr2=0x7f625405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 shutdown_connections 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:51.492+0000 7f625adc5700 1 -- 172.21.15.118:0/2808426579 wait complete. 2024-02-23T00:34:51.494 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:52.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:50.353082+0000 mgr.smithi090.pxxmlz (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:52.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:51.491057+0000 mon.smithi090 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.118:0/2808426579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:53.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:53.155 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:54.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:52.353378+0000 mgr.smithi090.pxxmlz (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.239+0000 7f5604f64700 1 -- 172.21.15.118:0/2599221339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 msgr2=0x7f56000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.239+0000 7f5604f64700 1 --2- 172.21.15.118:0/2599221339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56000f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f55f0009fb0 tx=0x7f55f0009b30).stop 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.239+0000 7f5604f64700 1 -- 172.21.15.118:0/2599221339 shutdown_connections 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.239+0000 7f5604f64700 1 --2- 172.21.15.118:0/2599221339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56000f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.239+0000 7f5604f64700 1 -- 172.21.15.118:0/2599221339 >> 172.21.15.118:0/2599221339 conn(0x7f56000f0b70 msgr2=0x7f56000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.240+0000 7f5604f64700 1 -- 172.21.15.118:0/2599221339 shutdown_connections 2024-02-23T00:34:55.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.240+0000 7f5604f64700 1 -- 172.21.15.118:0/2599221339 wait complete. 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.240+0000 7f5604f64700 1 Processor -- start 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.240+0000 7f5604f64700 1 -- start start 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f5604f64700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f5604f64700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5600060240 con 0x7f56000f5410 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55fe59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55fe59c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53852/0 (socket says 172.21.15.118:53852) 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55fe59c700 1 -- 172.21.15.118:0/1595683552 learned_addr learned my addr 172.21.15.118:0/1595683552 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55fe59c700 1 -- 172.21.15.118:0/1595683552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f0005040 con 0x7f56000f5410 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55fe59c700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f55f0005010 tx=0x7f55f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f0009320 con 0x7f56000f5410 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55f0013380 con 0x7f56000f5410 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5600060440 con 0x7f56000f5410 2024-02-23T00:34:55.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55f0014e50 con 0x7f56000f5410 2024-02-23T00:34:55.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.241+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5600109b90 con 0x7f56000f5410 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.242+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55f00095f0 con 0x7f56000f5410 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.242+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5600045ad0 con 0x7f56000f5410 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.242+0000 7f55eeffd700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 0x7f55e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.242+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55f0040e90 con 0x7f56000f5410 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.243+0000 7f55fdd9b700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 0x7f55e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:55.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.243+0000 7f55fdd9b700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 0x7f55e80323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f55f40096a0 tx=0x7f55f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:55.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.247+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55f0014420 con 0x7f56000f5410 2024-02-23T00:34:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.454+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f560010a3b0 con 0x7f56000f5410 2024-02-23T00:34:55.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.455+0000 7f55eeffd700 1 -- 172.21.15.118:0/1595683552 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55f0010230 con 0x7f56000f5410 2024-02-23T00:34:55.457 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:55.457 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.456+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 msgr2=0x7f55e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.456+0000 7f5604f64700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 0x7f55e80323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f55f40096a0 tx=0x7f55f4006b40).stop 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.456+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 msgr2=0x7f56001094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f55f0005010 tx=0x7f55f000b040).stop 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 shutdown_connections 2024-02-23T00:34:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f55e802ff10 0x7f55e80323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 --2- 172.21.15.118:0/1595683552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f56000f5410 0x7f56001094e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 >> 172.21.15.118:0/1595683552 conn(0x7f56000f0b70 msgr2=0x7f56000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 shutdown_connections 2024-02-23T00:34:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:55.457+0000 7f5604f64700 1 -- 172.21.15.118:0/1595683552 wait complete. 2024-02-23T00:34:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:34:56.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:54.353712+0000 mgr.smithi090.pxxmlz (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:56.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:55.455812+0000 mon.smithi090 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.118:0/1595683552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:34:57.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:34:57.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:34:58.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:56.354097+0000 mgr.smithi090.pxxmlz (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:34:59.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 -- 172.21.15.118:0/1976241703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 msgr2=0x7f74e80f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:59.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 --2- 172.21.15.118:0/1976241703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e80f5750 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f74d8009fb0 tx=0x7f74d8009b30).stop 2024-02-23T00:34:59.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 -- 172.21.15.118:0/1976241703 shutdown_connections 2024-02-23T00:34:59.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 --2- 172.21.15.118:0/1976241703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e80f5750 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:59.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 -- 172.21.15.118:0/1976241703 >> 172.21.15.118:0/1976241703 conn(0x7f74e80f0a90 msgr2=0x7f74e80f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:59.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 -- 172.21.15.118:0/1976241703 shutdown_connections 2024-02-23T00:34:59.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.251+0000 7f74f01f4700 1 -- 172.21.15.118:0/1976241703 wait complete. 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.252+0000 7f74f01f4700 1 Processor -- start 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.252+0000 7f74f01f4700 1 -- start start 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.252+0000 7f74f01f4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.252+0000 7f74f01f4700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e8105e70 con 0x7f74e80f5330 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74edf90700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74edf90700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53868/0 (socket says 172.21.15.118:53868) 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74edf90700 1 -- 172.21.15.118:0/851469390 learned_addr learned my addr 172.21.15.118:0/851469390 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74edf90700 1 -- 172.21.15.118:0/851469390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d8005040 con 0x7f74e80f5330 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74edf90700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f74d8005010 tx=0x7f74d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d8009320 con 0x7f74e80f5330 2024-02-23T00:34:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74d8013380 con 0x7f74e80f5330 2024-02-23T00:34:59.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e8106070 con 0x7f74e80f5330 2024-02-23T00:34:59.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d8014e50 con 0x7f74e80f5330 2024-02-23T00:34:59.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.253+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e8106500 con 0x7f74e80f5330 2024-02-23T00:34:59.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.254+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74d8014420 con 0x7f74e80f5330 2024-02-23T00:34:59.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.254+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e8045ad0 con 0x7f74e80f5330 2024-02-23T00:34:59.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.254+0000 7f74de7fc700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 0x7f74d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:59.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.254+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74d8040f00 con 0x7f74e80f5330 2024-02-23T00:34:59.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.254+0000 7f74ed78f700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 0x7f74d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:59.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.255+0000 7f74ed78f700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 0x7f74d40323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f74e40096a0 tx=0x7f74e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:59.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.259+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74d80095f0 con 0x7f74e80f5330 2024-02-23T00:34:59.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.464+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74e805ea50 con 0x7f74e80f5330 2024-02-23T00:34:59.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.465+0000 7f74de7fc700 1 -- 172.21.15.118:0/851469390 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74d801db70 con 0x7f74e80f5330 2024-02-23T00:34:59.466 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:34:59.466 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 msgr2=0x7f74d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 0x7f74d40323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f74e40096a0 tx=0x7f74e4006b40).stop 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 msgr2=0x7f74e8105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f74d8005010 tx=0x7f74d800b040).stop 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 shutdown_connections 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f74d402ff10 0x7f74d40323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 --2- 172.21.15.118:0/851469390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74e80f5330 0x7f74e8105980 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 >> 172.21.15.118:0/851469390 conn(0x7f74e80f0a90 msgr2=0x7f74e80f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 shutdown_connections 2024-02-23T00:34:59.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:34:59.466+0000 7f74f01f4700 1 -- 172.21.15.118:0/851469390 wait complete. 2024-02-23T00:34:59.469 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:00.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:34:58.354396+0000 mgr.smithi090.pxxmlz (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:00.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:34:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:34:59.465544+0000 mon.smithi090 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.118:0/851469390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:01.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:01.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:02.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:00.354660+0000 mgr.smithi090.pxxmlz (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- 172.21.15.118:0/312950871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 msgr2=0x7f8ab40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/312950871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab40f4be0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f8aa8009fb0 tx=0x7f8aa8009b30).stop 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- 172.21.15.118:0/312950871 shutdown_connections 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/312950871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab40f4be0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- 172.21.15.118:0/312950871 >> 172.21.15.118:0/312950871 conn(0x7f8ab40f02e0 msgr2=0x7f8ab40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- 172.21.15.118:0/312950871 shutdown_connections 2024-02-23T00:35:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- 172.21.15.118:0/312950871 wait complete. 2024-02-23T00:35:03.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 Processor -- start 2024-02-23T00:35:03.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.277+0000 7f8aba9e7700 1 -- start start 2024-02-23T00:35:03.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8aba9e7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:03.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8aba9e7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4106cc0 con 0x7f8ab40f47c0 2024-02-23T00:35:03.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:03.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:53892/0 (socket says 172.21.15.118:53892) 2024-02-23T00:35:03.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab3fff700 1 -- 172.21.15.118:0/385557819 learned_addr learned my addr 172.21.15.118:0/385557819 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab3fff700 1 -- 172.21.15.118:0/385557819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8005040 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab3fff700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f8aa8005010 tx=0x7f8aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.278+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa8009320 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8aa8013380 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab4106ec0 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa8014e50 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab4107350 con 0x7f8ab40f47c0 2024-02-23T00:35:03.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8aa80263a0 con 0x7f8ab40f47c0 2024-02-23T00:35:03.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.279+0000 7f8ab0ff9700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 0x7f8a9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:03.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.280+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8aa8040e90 con 0x7f8ab40f47c0 2024-02-23T00:35:03.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.280+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab4045ad0 con 0x7f8ab40f47c0 2024-02-23T00:35:03.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.280+0000 7f8ab37fe700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 0x7f8a9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:03.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.280+0000 7f8ab37fe700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 0x7f8a9c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f8aa40096a0 tx=0x7f8aa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:03.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.285+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8aa8012920 con 0x7f8ab40f47c0 2024-02-23T00:35:03.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.488+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ab4108070 con 0x7f8ab40f47c0 2024-02-23T00:35:03.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.488+0000 7f8ab0ff9700 1 -- 172.21.15.118:0/385557819 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8aa8012330 con 0x7f8ab40f47c0 2024-02-23T00:35:03.490 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:03.491 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:03.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 msgr2=0x7f8a9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:03.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 0x7f8a9c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f8aa40096a0 tx=0x7f8aa4006b40).stop 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 msgr2=0x7f8ab41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f8aa8005010 tx=0x7f8aa800b040).stop 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 shutdown_connections 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f8a9c02ff10 0x7f8a9c0323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 --2- 172.21.15.118:0/385557819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ab40f47c0 0x7f8ab41067d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 >> 172.21.15.118:0/385557819 conn(0x7f8ab40f02e0 msgr2=0x7f8ab40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 shutdown_connections 2024-02-23T00:35:03.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:03.490+0000 7f8aba9e7700 1 -- 172.21.15.118:0/385557819 wait complete. 2024-02-23T00:35:03.494 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:04.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:02.354986+0000 mgr.smithi090.pxxmlz (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:04.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:03.489223+0000 mon.smithi090 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.118:0/385557819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:05.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:05.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:06.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:04.355365+0000 mgr.smithi090.pxxmlz (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:07.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- 172.21.15.118:0/2135539479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 msgr2=0x7fa0400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 --2- 172.21.15.118:0/2135539479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa0400f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa030009fb0 tx=0x7fa030009b30).stop 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- 172.21.15.118:0/2135539479 shutdown_connections 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 --2- 172.21.15.118:0/2135539479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa0400f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- 172.21.15.118:0/2135539479 >> 172.21.15.118:0/2135539479 conn(0x7fa0400f0b70 msgr2=0x7fa0400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- 172.21.15.118:0/2135539479 shutdown_connections 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- 172.21.15.118:0/2135539479 wait complete. 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 Processor -- start 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.233+0000 7fa045b6a700 1 -- start start 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa045b6a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa045b6a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0401072f0 con 0x7fa0400f5410 2024-02-23T00:35:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa03f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa03f7fe700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:56236/0 (socket says 172.21.15.118:56236) 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa03f7fe700 1 -- 172.21.15.118:0/3760272984 learned_addr learned my addr 172.21.15.118:0/3760272984 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa03f7fe700 1 -- 172.21.15.118:0/3760272984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa030005040 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa03f7fe700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa030005010 tx=0x7fa03000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa030009320 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa040060220 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa030013380 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa030014e50 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.234+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04010bdf0 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.235+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0300095f0 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.235+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa040045ad0 con 0x7fa0400f5410 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.235+0000 7fa037fff700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 0x7fa02c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:07.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.235+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa030040fa0 con 0x7fa0400f5410 2024-02-23T00:35:07.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.236+0000 7fa03effd700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 0x7fa02c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:07.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.236+0000 7fa03effd700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 0x7fa02c0323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0280096a0 tx=0x7fa028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:07.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.240+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa030014420 con 0x7fa0400f5410 2024-02-23T00:35:07.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.447+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa04010c5b0 con 0x7fa0400f5410 2024-02-23T00:35:07.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.447+0000 7fa037fff700 1 -- 172.21.15.118:0/3760272984 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa030010230 con 0x7fa0400f5410 2024-02-23T00:35:07.450 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:07.450 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 msgr2=0x7fa02c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 0x7fa02c0323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0280096a0 tx=0x7fa028006b40).stop 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 msgr2=0x7fa04010b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa030005010 tx=0x7fa03000b040).stop 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 shutdown_connections 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fa02c02ff10 0x7fa02c0323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 --2- 172.21.15.118:0/3760272984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0400f5410 0x7fa04010b900 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 >> 172.21.15.118:0/3760272984 conn(0x7fa0400f0b70 msgr2=0x7fa0400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:07.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 shutdown_connections 2024-02-23T00:35:07.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:07.449+0000 7fa045b6a700 1 -- 172.21.15.118:0/3760272984 wait complete. 2024-02-23T00:35:07.453 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:08.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:06.355782+0000 mgr.smithi090.pxxmlz (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:08.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:07.448371+0000 mon.smithi090 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.118:0/3760272984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:09.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:09.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:10.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:08.356195+0000 mgr.smithi090.pxxmlz (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:11.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- 172.21.15.118:0/622136601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 msgr2=0x7fc3c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/622136601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fc3b4009fb0 tx=0x7fc3b4009b30).stop 2024-02-23T00:35:11.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- 172.21.15.118:0/622136601 shutdown_connections 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/622136601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- 172.21.15.118:0/622136601 >> 172.21.15.118:0/622136601 conn(0x7fc3c40f0b70 msgr2=0x7fc3c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- 172.21.15.118:0/622136601 shutdown_connections 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- 172.21.15.118:0/622136601 wait complete. 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 Processor -- start 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.334+0000 7fc3ca88b700 1 -- start start 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3ca88b700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3ca88b700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c40fb960 con 0x7fc3c40f5410 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c3fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:56248/0 (socket says 172.21.15.118:56248) 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c3fff700 1 -- 172.21.15.118:0/2870578302 learned_addr learned my addr 172.21.15.118:0/2870578302 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c3fff700 1 -- 172.21.15.118:0/2870578302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b4005040 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c3fff700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fc3b4005010 tx=0x7fc3b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3b4009320 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c40fbb00 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3b4013380 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3b4014e50 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.335+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c40fbf90 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.336+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3b40095f0 con 0x7fc3c40f5410 2024-02-23T00:35:11.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.336+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c4059100 con 0x7fc3c40f5410 2024-02-23T00:35:11.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.336+0000 7fc3c0ff9700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 0x7fc3ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.336+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3b4040fa0 con 0x7fc3c40f5410 2024-02-23T00:35:11.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.337+0000 7fc3c37fe700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 0x7fc3ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.337+0000 7fc3c37fe700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 0x7fc3ac0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc3b80096a0 tx=0x7fc3b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:11.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.341+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3b4014420 con 0x7fc3c40f5410 2024-02-23T00:35:11.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.544+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3c40fafa0 con 0x7fc3c40f5410 2024-02-23T00:35:11.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.545+0000 7fc3c0ff9700 1 -- 172.21.15.118:0/2870578302 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3b4010230 con 0x7fc3c40f5410 2024-02-23T00:35:11.547 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:11.547 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 msgr2=0x7fc3ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 0x7fc3ac0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc3b80096a0 tx=0x7fc3b8006b40).stop 2024-02-23T00:35:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 msgr2=0x7fc3c40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fc3b4005010 tx=0x7fc3b400b040).stop 2024-02-23T00:35:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 shutdown_connections 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc3ac02ff10 0x7fc3ac0323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.546+0000 7fc3ca88b700 1 --2- 172.21.15.118:0/2870578302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c40f5410 0x7fc3c40fd4e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.547+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 >> 172.21.15.118:0/2870578302 conn(0x7fc3c40f0b70 msgr2=0x7fc3c405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.547+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 shutdown_connections 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:11.547+0000 7fc3ca88b700 1 -- 172.21.15.118:0/2870578302 wait complete. 2024-02-23T00:35:11.549 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:12.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:10.356546+0000 mgr.smithi090.pxxmlz (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:12.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:11.546085+0000 mon.smithi090 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.118:0/2870578302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:13.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:13.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:14.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:12.356905+0000 mgr.smithi090.pxxmlz (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 -- 172.21.15.118:0/167170660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 msgr2=0x7f39240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 --2- 172.21.15.118:0/167170660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f39240f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f3920009fb0 tx=0x7f3920009b30).stop 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 -- 172.21.15.118:0/167170660 shutdown_connections 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 --2- 172.21.15.118:0/167170660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f39240f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 -- 172.21.15.118:0/167170660 >> 172.21.15.118:0/167170660 conn(0x7f39240f1c70 msgr2=0x7f39240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 -- 172.21.15.118:0/167170660 shutdown_connections 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.353+0000 7f392c213700 1 -- 172.21.15.118:0/167170660 wait complete. 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f392c213700 1 Processor -- start 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f392c213700 1 -- start start 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f392c213700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:15.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f392c213700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3924069e20 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f3929faf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f3929faf700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39814/0 (socket says 172.21.15.118:39814) 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f3929faf700 1 -- 172.21.15.118:0/2408658682 learned_addr learned my addr 172.21.15.118:0/2408658682 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.354+0000 7f3929faf700 1 -- 172.21.15.118:0/2408658682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920005040 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f3929faf700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f3920005010 tx=0x7f392000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3920009320 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3924069fc0 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3920013380 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3920014e50 con 0x7f39240f6510 2024-02-23T00:35:15.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.355+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392406a450 con 0x7f39240f6510 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.356+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39200095f0 con 0x7f39240f6510 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.356+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924059100 con 0x7f39240f6510 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.356+0000 7f391a7fc700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 0x7f39100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.356+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3920040e90 con 0x7f39240f6510 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.356+0000 7f39297ae700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 0x7f39100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:15.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.357+0000 7f39297ae700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 0x7f39100323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f39140096a0 tx=0x7f3914006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:15.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.360+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3920014420 con 0x7f39240f6510 2024-02-23T00:35:15.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.564+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39240fc0a0 con 0x7f39240f6510 2024-02-23T00:35:15.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.565+0000 7f391a7fc700 1 -- 172.21.15.118:0/2408658682 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3920010230 con 0x7f39240f6510 2024-02-23T00:35:15.567 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:15.567 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:15.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.566+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 msgr2=0x7f39100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:15.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.566+0000 7f392c213700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 0x7f39100323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f39140096a0 tx=0x7f3914006b40).stop 2024-02-23T00:35:15.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.566+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 msgr2=0x7f392406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:15.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.566+0000 7f392c213700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f3920005010 tx=0x7f392000b040).stop 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 shutdown_connections 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f391002ff10 0x7f39100323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 --2- 172.21.15.118:0/2408658682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f39240f6510 0x7f392406b9a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 >> 172.21.15.118:0/2408658682 conn(0x7f39240f1c70 msgr2=0x7f39240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 shutdown_connections 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:15.567+0000 7f392c213700 1 -- 172.21.15.118:0/2408658682 wait complete. 2024-02-23T00:35:15.569 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:16.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:14.357188+0000 mgr.smithi090.pxxmlz (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:17.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:16 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:15.566037+0000 mon.smithi090 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.118:0/2408658682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:17.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:17.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:18.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:16.357461+0000 mgr.smithi090.pxxmlz (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:19.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- 172.21.15.118:0/243358562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 msgr2=0x7fcfc80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:19.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/243358562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc80f90f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fcfc4009fb0 tx=0x7fcfc4009b30).stop 2024-02-23T00:35:19.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- 172.21.15.118:0/243358562 shutdown_connections 2024-02-23T00:35:19.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/243358562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc80f90f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:19.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- 172.21.15.118:0/243358562 >> 172.21.15.118:0/243358562 conn(0x7fcfc80f0b70 msgr2=0x7fcfc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:19.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- 172.21.15.118:0/243358562 shutdown_connections 2024-02-23T00:35:19.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- 172.21.15.118:0/243358562 wait complete. 2024-02-23T00:35:19.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 Processor -- start 2024-02-23T00:35:19.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- start start 2024-02-23T00:35:19.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:19.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.371+0000 7fcfcfbab700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc8106c70 con 0x7fcfc8060280 2024-02-23T00:35:19.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcd947700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:19.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcd947700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39832/0 (socket says 172.21.15.118:39832) 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcd947700 1 -- 172.21.15.118:0/4257242383 learned_addr learned my addr 172.21.15.118:0/4257242383 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcd947700 1 -- 172.21.15.118:0/4257242383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4005040 con 0x7fcfc8060280 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcd947700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fcfc4009830 tx=0x7fcfc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfc4009320 con 0x7fcfc8060280 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc8106e70 con 0x7fcfc8060280 2024-02-23T00:35:19.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfc4013380 con 0x7fcfc8060280 2024-02-23T00:35:19.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfc4014420 con 0x7fcfc8060280 2024-02-23T00:35:19.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.372+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc8107300 con 0x7fcfc8060280 2024-02-23T00:35:19.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.373+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfc401e420 con 0x7fcfc8060280 2024-02-23T00:35:19.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.373+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc8045ad0 con 0x7fcfc8060280 2024-02-23T00:35:19.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.373+0000 7fcfbe7fc700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 0x7fcfb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:19.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.373+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfc4040e20 con 0x7fcfc8060280 2024-02-23T00:35:19.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.374+0000 7fcfcd146700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 0x7fcfb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:19.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.374+0000 7fcfcd146700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 0x7fcfb4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fcfb80096a0 tx=0x7fcfb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:19.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.378+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfc401c970 con 0x7fcfc8060280 2024-02-23T00:35:19.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.582+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfc80fc090 con 0x7fcfc8060280 2024-02-23T00:35:19.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.583+0000 7fcfbe7fc700 1 -- 172.21.15.118:0/4257242383 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfc402c2d0 con 0x7fcfc8060280 2024-02-23T00:35:19.585 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:19.585 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 msgr2=0x7fcfb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 0x7fcfb4032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fcfb80096a0 tx=0x7fcfb8006b40).stop 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 msgr2=0x7fcfc8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fcfc4009830 tx=0x7fcfc400b040).stop 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 shutdown_connections 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcfb402ff60 0x7fcfb4032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 --2- 172.21.15.118:0/4257242383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfc8060280 0x7fcfc8106780 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:19.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 >> 172.21.15.118:0/4257242383 conn(0x7fcfc80f0b70 msgr2=0x7fcfc80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:19.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 shutdown_connections 2024-02-23T00:35:19.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:19.584+0000 7fcfcfbab700 1 -- 172.21.15.118:0/4257242383 wait complete. 2024-02-23T00:35:19.589 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:20.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:18.357818+0000 mgr.smithi090.pxxmlz (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:21.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:20 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:19.584853+0000 mon.smithi090 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.118:0/4257242383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:21.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:21.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:22.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:20.358152+0000 mgr.smithi090.pxxmlz (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.535+0000 7f6dab581700 1 -- 172.21.15.118:0/909138136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 msgr2=0x7f6da40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.535+0000 7f6dab581700 1 --2- 172.21.15.118:0/909138136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da40f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6d98009fb0 tx=0x7f6d98009b30).stop 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- 172.21.15.118:0/909138136 shutdown_connections 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 --2- 172.21.15.118:0/909138136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da40f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- 172.21.15.118:0/909138136 >> 172.21.15.118:0/909138136 conn(0x7f6da40f0b70 msgr2=0x7f6da40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:23.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- 172.21.15.118:0/909138136 shutdown_connections 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- 172.21.15.118:0/909138136 wait complete. 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 Processor -- start 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- start start 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.536+0000 7f6dab581700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da4060240 con 0x7f6da40f5410 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6da931d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6da931d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:39850/0 (socket says 172.21.15.118:39850) 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6da931d700 1 -- 172.21.15.118:0/2894515593 learned_addr learned my addr 172.21.15.118:0/2894515593 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6da931d700 1 -- 172.21.15.118:0/2894515593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98005040 con 0x7f6da40f5410 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6da931d700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6d98005010 tx=0x7f6d9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d98009320 con 0x7f6da40f5410 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da4060440 con 0x7f6da40f5410 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d98013380 con 0x7f6da40f5410 2024-02-23T00:35:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d98014e50 con 0x7f6da40f5410 2024-02-23T00:35:23.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.537+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da4109b90 con 0x7f6da40f5410 2024-02-23T00:35:23.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.538+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d980095f0 con 0x7f6da40f5410 2024-02-23T00:35:23.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.538+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da4045ad0 con 0x7f6da40f5410 2024-02-23T00:35:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.538+0000 7f6d95ffb700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 0x7f6d900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.538+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d98040ef0 con 0x7f6da40f5410 2024-02-23T00:35:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.539+0000 7f6da8b1c700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 0x7f6d900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.539+0000 7f6da8b1c700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 0x7f6d900323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6da00096a0 tx=0x7f6da0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:23.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.543+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d98014420 con 0x7f6da40f5410 2024-02-23T00:35:23.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.750+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6da410a3a0 con 0x7f6da40f5410 2024-02-23T00:35:23.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.750+0000 7f6d95ffb700 1 -- 172.21.15.118:0/2894515593 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d98010230 con 0x7f6da40f5410 2024-02-23T00:35:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:23.753 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 msgr2=0x7f6d900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 0x7f6d900323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6da00096a0 tx=0x7f6da0006b40).stop 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 msgr2=0x7f6da41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6d98005010 tx=0x7f6d9800b040).stop 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 shutdown_connections 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f6d9002ff10 0x7f6d900323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 --2- 172.21.15.118:0/2894515593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6da40f5410 0x7f6da41094e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 >> 172.21.15.118:0/2894515593 conn(0x7f6da40f0b70 msgr2=0x7f6da40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:23.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 shutdown_connections 2024-02-23T00:35:23.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:23.752+0000 7f6dab581700 1 -- 172.21.15.118:0/2894515593 wait complete. 2024-02-23T00:35:23.756 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:24.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:22.358460+0000 mgr.smithi090.pxxmlz (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:25.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:24 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:23.752415+0000 mon.smithi090 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.118:0/2894515593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:25.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:25.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:24.358786+0000 mgr.smithi090.pxxmlz (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:25.255476+0000 mon.smithi090 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:35:26.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:25.261150+0000 mon.smithi090 (mon.0) 326 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- 172.21.15.118:0/2038544560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 msgr2=0x7fe8640f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 --2- 172.21.15.118:0/2038544560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8640f8f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe854009fb0 tx=0x7fe854009b30).stop 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- 172.21.15.118:0/2038544560 shutdown_connections 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 --2- 172.21.15.118:0/2038544560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8640f8f80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- 172.21.15.118:0/2038544560 >> 172.21.15.118:0/2038544560 conn(0x7fe8640f09a0 msgr2=0x7fe8640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- 172.21.15.118:0/2038544560 shutdown_connections 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- 172.21.15.118:0/2038544560 wait complete. 2024-02-23T00:35:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 Processor -- start 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.502+0000 7fe863fff700 1 -- start start 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe863fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe863fff700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864106ad0 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe862ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe862ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60560/0 (socket says 172.21.15.118:60560) 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe862ffd700 1 -- 172.21.15.118:0/3198156640 learned_addr learned my addr 172.21.15.118:0/3198156640 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe862ffd700 1 -- 172.21.15.118:0/3198156640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854005040 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe862ffd700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe854005010 tx=0x7fe85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe854009320 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe864106cd0 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe864107160 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe854013380 con 0x7fe864060180 2024-02-23T00:35:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.503+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe854014df0 con 0x7fe864060180 2024-02-23T00:35:27.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.504+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844004fa0 con 0x7fe864060180 2024-02-23T00:35:27.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.504+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8540095f0 con 0x7fe864060180 2024-02-23T00:35:27.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.504+0000 7fe84b7fe700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 0x7fe84c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:27.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.504+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe854040e00 con 0x7fe864060180 2024-02-23T00:35:27.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.505+0000 7fe8627fc700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 0x7fe84c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:27.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.505+0000 7fe8627fc700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 0x7fe84c0323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe8580096a0 tx=0x7fe858006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:27.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.509+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe854012980 con 0x7fe864060180 2024-02-23T00:35:27.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.713+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe844005670 con 0x7fe864060180 2024-02-23T00:35:27.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.714+0000 7fe84b7fe700 1 -- 172.21.15.118:0/3198156640 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe854012980 con 0x7fe864060180 2024-02-23T00:35:27.715 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:27.715 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 msgr2=0x7fe84c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 0x7fe84c0323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe8580096a0 tx=0x7fe858006b40).stop 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 msgr2=0x7fe8641065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe854005010 tx=0x7fe85400b040).stop 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 shutdown_connections 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fe84c02ff10 0x7fe84c0323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 --2- 172.21.15.118:0/3198156640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe864060180 0x7fe8641065e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 >> 172.21.15.118:0/3198156640 conn(0x7fe8640f09a0 msgr2=0x7fe8640f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 shutdown_connections 2024-02-23T00:35:27.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:27.715+0000 7fe863fff700 1 -- 172.21.15.118:0/3198156640 wait complete. 2024-02-23T00:35:27.718 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:28.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:27 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:26.359175+0000 mgr.smithi090.pxxmlz (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:29.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:28 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:27.714418+0000 mon.smithi090 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.118:0/3198156640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:29.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:29.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:30.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:29 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:28.359606+0000 mgr.smithi090.pxxmlz (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- 172.21.15.118:0/1503721179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 msgr2=0x7f130c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 --2- 172.21.15.118:0/1503721179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f1300009fb0 tx=0x7f1300009b30).stop 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- 172.21.15.118:0/1503721179 shutdown_connections 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 --2- 172.21.15.118:0/1503721179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- 172.21.15.118:0/1503721179 >> 172.21.15.118:0/1503721179 conn(0x7f130c0f0b70 msgr2=0x7f130c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- 172.21.15.118:0/1503721179 shutdown_connections 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- 172.21.15.118:0/1503721179 wait complete. 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 Processor -- start 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.597+0000 7f1312df7700 1 -- start start 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1312df7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1312df7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c0fb960 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1310b93700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1310b93700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60582/0 (socket says 172.21.15.118:60582) 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1310b93700 1 -- 172.21.15.118:0/893693148 learned_addr learned my addr 172.21.15.118:0/893693148 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1310b93700 1 -- 172.21.15.118:0/893693148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300005040 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1310b93700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f13000089f0 tx=0x7f130000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1300009320 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c0fbb00 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1300013380 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130001ed80 con 0x7f130c0f5410 2024-02-23T00:35:31.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.598+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c0fbf90 con 0x7f130c0f5410 2024-02-23T00:35:31.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.599+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f130001e420 con 0x7f130c0f5410 2024-02-23T00:35:31.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.599+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c059100 con 0x7f130c0f5410 2024-02-23T00:35:31.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.599+0000 7f13097fa700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 0x7f12fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:31.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.599+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13000410d0 con 0x7f130c0f5410 2024-02-23T00:35:31.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.599+0000 7f130bfff700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 0x7f12fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:31.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.600+0000 7f130bfff700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 0x7f12fc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f12f80096a0 tx=0x7f12f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:31.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.604+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f130001cb10 con 0x7f130c0f5410 2024-02-23T00:35:31.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.808+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f130c0fafa0 con 0x7f130c0f5410 2024-02-23T00:35:31.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.808+0000 7f13097fa700 1 -- 172.21.15.118:0/893693148 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1300004490 con 0x7f130c0f5410 2024-02-23T00:35:31.810 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:31.810 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:31.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 msgr2=0x7f12fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 0x7f12fc032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f12f80096a0 tx=0x7f12f8006b40).stop 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 msgr2=0x7f130c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f13000089f0 tx=0x7f130000b040).stop 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 shutdown_connections 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f12fc02ff60 0x7f12fc032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 --2- 172.21.15.118:0/893693148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f130c0f5410 0x7f130c0fd4e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 >> 172.21.15.118:0/893693148 conn(0x7f130c0f0b70 msgr2=0x7f130c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 shutdown_connections 2024-02-23T00:35:31.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:31.810+0000 7f1312df7700 1 -- 172.21.15.118:0/893693148 wait complete. 2024-02-23T00:35:31.813 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:32.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:31 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:30.360075+0000 mgr.smithi090.pxxmlz (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:33.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:32 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:31.809291+0000 mon.smithi090 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.118:0/893693148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:33.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:33.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:34.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:33 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:32.360525+0000 mgr.smithi090.pxxmlz (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:35.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- 172.21.15.118:0/2878400918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 msgr2=0x7f7a580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/2878400918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a580f55c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f7a54009fb0 tx=0x7f7a54009b30).stop 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- 172.21.15.118:0/2878400918 shutdown_connections 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/2878400918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a580f55c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- 172.21.15.118:0/2878400918 >> 172.21.15.118:0/2878400918 conn(0x7f7a580f09a0 msgr2=0x7f7a580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- 172.21.15.118:0/2878400918 shutdown_connections 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- 172.21.15.118:0/2878400918 wait complete. 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 Processor -- start 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.775+0000 7f7a5ed56700 1 -- start start 2024-02-23T00:35:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5ed56700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5ed56700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a581059c0 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5dd54700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5dd54700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59600/0 (socket says 172.21.15.118:59600) 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5dd54700 1 -- 172.21.15.118:0/1161055873 learned_addr learned my addr 172.21.15.118:0/1161055873 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5dd54700 1 -- 172.21.15.118:0/1161055873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54005040 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5dd54700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f7a540089f0 tx=0x7f7a5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a54009320 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58105bc0 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a54013380 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a54014e20 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.776+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a58106050 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.777+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a540095f0 con 0x7f7a580f51a0 2024-02-23T00:35:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.777+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a58045ad0 con 0x7f7a580f51a0 2024-02-23T00:35:35.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.777+0000 7f7a4e7fc700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:35.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.778+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a54040f50 con 0x7f7a580f51a0 2024-02-23T00:35:35.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.778+0000 7f7a5d553700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:35.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.778+0000 7f7a5d553700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 0x7f7a44032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7a480096a0 tx=0x7f7a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.782+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a54014420 con 0x7f7a580f51a0 2024-02-23T00:35:35.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.989+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a58106f00 con 0x7f7a580f51a0 2024-02-23T00:35:35.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.990+0000 7f7a4e7fc700 1 -- 172.21.15.118:0/1161055873 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a5401d980 con 0x7f7a580f51a0 2024-02-23T00:35:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:35.992 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 msgr2=0x7f7a44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 0x7f7a44032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7a480096a0 tx=0x7f7a48006b40).stop 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 msgr2=0x7f7a581054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f7a540089f0 tx=0x7f7a5400b040).stop 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 shutdown_connections 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f7a4402ff60 0x7f7a44032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 --2- 172.21.15.118:0/1161055873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a580f51a0 0x7f7a581054d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 >> 172.21.15.118:0/1161055873 conn(0x7f7a580f09a0 msgr2=0x7f7a580f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 shutdown_connections 2024-02-23T00:35:35.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:35.992+0000 7f7a5ed56700 1 -- 172.21.15.118:0/1161055873 wait complete. 2024-02-23T00:35:35.995 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:36.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:35 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:34.360879+0000 mgr.smithi090.pxxmlz (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:37.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:36 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:35.991079+0000 mon.smithi090 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.118:0/1161055873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:37.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:37.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:38.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:37 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:36.361156+0000 mgr.smithi090.pxxmlz (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:40.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:39 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:38.361502+0000 mgr.smithi090.pxxmlz (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:40.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 -- 172.21.15.118:0/255214892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 msgr2=0x7f846c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 --2- 172.21.15.118:0/255214892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f8460009fb0 tx=0x7f8460009b30).stop 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 -- 172.21.15.118:0/255214892 shutdown_connections 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 --2- 172.21.15.118:0/255214892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 -- 172.21.15.118:0/255214892 >> 172.21.15.118:0/255214892 conn(0x7f846c0f0b70 msgr2=0x7f846c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 -- 172.21.15.118:0/255214892 shutdown_connections 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.011+0000 7f84710c7700 1 -- 172.21.15.118:0/255214892 wait complete. 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f84710c7700 1 Processor -- start 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f84710c7700 1 -- start start 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f84710c7700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:40.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f84710c7700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c106cd0 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f846ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f846ad9d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59618/0 (socket says 172.21.15.118:59618) 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f846ad9d700 1 -- 172.21.15.118:0/782866838 learned_addr learned my addr 172.21.15.118:0/782866838 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.012+0000 7f846ad9d700 1 -- 172.21.15.118:0/782866838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460005040 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f846ad9d700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8460005010 tx=0x7f846000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8460009320 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c106ed0 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8460013380 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8460014e50 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c107360 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84600095f0 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.013+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c045ad0 con 0x7f846c0f5410 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.014+0000 7f84537fe700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:40.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.014+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8460040fa0 con 0x7f846c0f5410 2024-02-23T00:35:40.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.014+0000 7f846a59c700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:40.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.014+0000 7f846a59c700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 0x7f84540323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f845c0096a0 tx=0x7f845c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:40.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.018+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8460014420 con 0x7f846c0f5410 2024-02-23T00:35:40.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.221+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f846c108210 con 0x7f846c0f5410 2024-02-23T00:35:40.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.222+0000 7f84537fe700 1 -- 172.21.15.118:0/782866838 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8460010230 con 0x7f846c0f5410 2024-02-23T00:35:40.223 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:40.223 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 msgr2=0x7f84540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 0x7f84540323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f845c0096a0 tx=0x7f845c006b40).stop 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 msgr2=0x7f846c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8460005010 tx=0x7f846000b040).stop 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 shutdown_connections 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 --2- 172.21.15.118:0/782866838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f846c0f5410 0x7f846c1067e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:40.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 >> 172.21.15.118:0/782866838 conn(0x7f846c0f0b70 msgr2=0x7f846c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:40.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 shutdown_connections 2024-02-23T00:35:40.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:40.223+0000 7f84710c7700 1 -- 172.21.15.118:0/782866838 wait complete. 2024-02-23T00:35:40.226 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:41.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:40 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:40.222816+0000 mon.smithi090 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.118:0/782866838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:41.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:41.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:42.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:41 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:40.361897+0000 mgr.smithi090.pxxmlz (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:44.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:43 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:42.362194+0000 mgr.smithi090.pxxmlz (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:44.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 -- 172.21.15.118:0/2363350027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 msgr2=0x7f9c1c0f5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2363350027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c0f5b90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f9c10009fb0 tx=0x7f9c10009b30).stop 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 -- 172.21.15.118:0/2363350027 shutdown_connections 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2363350027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c0f5b90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 -- 172.21.15.118:0/2363350027 >> 172.21.15.118:0/2363350027 conn(0x7f9c1c0f0120 msgr2=0x7f9c1c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 -- 172.21.15.118:0/2363350027 shutdown_connections 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.012+0000 7f9c21d86700 1 -- 172.21.15.118:0/2363350027 wait complete. 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c21d86700 1 Processor -- start 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c21d86700 1 -- start start 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c21d86700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c21d86700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c105970 con 0x7f9c1c0f5770 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c20d84700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:44.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c20d84700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:59638/0 (socket says 172.21.15.118:59638) 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.013+0000 7f9c20d84700 1 -- 172.21.15.118:0/2313577645 learned_addr learned my addr 172.21.15.118:0/2313577645 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c20d84700 1 -- 172.21.15.118:0/2313577645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10005040 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c20d84700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f9c100089f0 tx=0x7f9c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10009320 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c10013380 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c105b70 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10014e20 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.014+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c106000 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.015+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c100095f0 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.015+0000 7f9c197fa700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 0x7f9c0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.015+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c10040f50 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.015+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c045ad0 con 0x7f9c1c0f5770 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.015+0000 7f9c1bfff700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 0x7f9c0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.016+0000 7f9c1bfff700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 0x7f9c0c0323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9c080096a0 tx=0x7f9c08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:44.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.020+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c10014420 con 0x7f9c1c0f5770 2024-02-23T00:35:44.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.226+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c1c106eb0 con 0x7f9c1c0f5770 2024-02-23T00:35:44.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.227+0000 7f9c197fa700 1 -- 172.21.15.118:0/2313577645 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c1001d980 con 0x7f9c1c0f5770 2024-02-23T00:35:44.228 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:44.229 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 msgr2=0x7f9c0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 0x7f9c0c0323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9c080096a0 tx=0x7f9c08006b40).stop 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 msgr2=0x7f9c1c105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f9c100089f0 tx=0x7f9c1000b040).stop 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 shutdown_connections 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f9c0c02ff10 0x7f9c0c0323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 --2- 172.21.15.118:0/2313577645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c1c0f5770 0x7f9c1c105480 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:44.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 >> 172.21.15.118:0/2313577645 conn(0x7f9c1c0f0120 msgr2=0x7f9c1c0f0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 shutdown_connections 2024-02-23T00:35:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:44.228+0000 7f9c21d86700 1 -- 172.21.15.118:0/2313577645 wait complete. 2024-02-23T00:35:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:45.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:44 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:44.227756+0000 mon.smithi090 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.118:0/2313577645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:46.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:45 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:44.362540+0000 mgr.smithi090.pxxmlz (mgr.14162) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:46.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:46.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:48.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:47 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:46.362841+0000 mgr.smithi090.pxxmlz (mgr.14162) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2788932283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 msgr2=0x7f1e980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2788932283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e980f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f1e88009fb0 tx=0x7f1e88009b30).stop 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2788932283 shutdown_connections 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2788932283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e980f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2788932283 >> 172.21.15.118:0/2788932283 conn(0x7f1e980f0b70 msgr2=0x7f1e980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2788932283 shutdown_connections 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2788932283 wait complete. 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 Processor -- start 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- start start 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:48.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.084+0000 7f1e9f7a0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e98060220 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9d53c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9d53c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55886/0 (socket says 172.21.15.118:55886) 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9d53c700 1 -- 172.21.15.118:0/2759050044 learned_addr learned my addr 172.21.15.118:0/2759050044 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9d53c700 1 -- 172.21.15.118:0/2759050044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88005040 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9d53c700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1e88005010 tx=0x7f1e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e88009320 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e98060420 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e88013380 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e88014e50 con 0x7f1e980f5410 2024-02-23T00:35:48.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.085+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e98109bb0 con 0x7f1e980f5410 2024-02-23T00:35:48.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.086+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e880263a0 con 0x7f1e980f5410 2024-02-23T00:35:48.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.086+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e98059100 con 0x7f1e980f5410 2024-02-23T00:35:48.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.086+0000 7f1e8dffb700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 0x7f1e840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:48.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.086+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e88041160 con 0x7f1e980f5410 2024-02-23T00:35:48.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.087+0000 7f1e9cd3b700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 0x7f1e840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:48.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.087+0000 7f1e9cd3b700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 0x7f1e840323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1e940096a0 tx=0x7f1e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:48.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.091+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e8801cb10 con 0x7f1e980f5410 2024-02-23T00:35:48.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.300+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e98045ad0 con 0x7f1e980f5410 2024-02-23T00:35:48.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.301+0000 7f1e8dffb700 1 -- 172.21.15.118:0/2759050044 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e8801cb10 con 0x7f1e980f5410 2024-02-23T00:35:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:48.303 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:48.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 msgr2=0x7f1e840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:48.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 0x7f1e840323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1e940096a0 tx=0x7f1e94006b40).stop 2024-02-23T00:35:48.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 msgr2=0x7f1e98109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:48.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1e88005010 tx=0x7f1e8800b040).stop 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 shutdown_connections 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1e8402ff10 0x7f1e840323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 --2- 172.21.15.118:0/2759050044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e980f5410 0x7f1e98109500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 >> 172.21.15.118:0/2759050044 conn(0x7f1e980f0b70 msgr2=0x7f1e980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 shutdown_connections 2024-02-23T00:35:48.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:48.302+0000 7f1e9f7a0700 1 -- 172.21.15.118:0/2759050044 wait complete. 2024-02-23T00:35:48.306 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:49.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:48 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:48.301613+0000 mon.smithi090 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.118:0/2759050044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:49.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:49.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:50.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:49 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:48.363097+0000 mgr.smithi090.pxxmlz (mgr.14162) 382 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:52.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:51 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:50.363374+0000 mgr.smithi090.pxxmlz (mgr.14162) 383 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 -- 172.21.15.118:0/2847893777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 msgr2=0x7fce040f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 --2- 172.21.15.118:0/2847893777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce040f6ee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fcdf4009fb0 tx=0x7fcdf4009b30).stop 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 -- 172.21.15.118:0/2847893777 shutdown_connections 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 --2- 172.21.15.118:0/2847893777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce040f6ee0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 -- 172.21.15.118:0/2847893777 >> 172.21.15.118:0/2847893777 conn(0x7fce040f0b70 msgr2=0x7fce040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 -- 172.21.15.118:0/2847893777 shutdown_connections 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.098+0000 7fce0b518700 1 -- 172.21.15.118:0/2847893777 wait complete. 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce0b518700 1 Processor -- start 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce0b518700 1 -- start start 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce0b518700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce0b518700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0410d2c0 con 0x7fce04060280 2024-02-23T00:35:52.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce092b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce092b4700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:55900/0 (socket says 172.21.15.118:55900) 2024-02-23T00:35:52.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.099+0000 7fce092b4700 1 -- 172.21.15.118:0/3084010854 learned_addr learned my addr 172.21.15.118:0/3084010854 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:52.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fce092b4700 1 -- 172.21.15.118:0/3084010854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4005040 con 0x7fce04060280 2024-02-23T00:35:52.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fce092b4700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fcdf40089f0 tx=0x7fcdf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf4009320 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0410d4c0 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdf4013380 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf4014e20 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.100+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0410e100 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.101+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0410d950 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.101+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdf40095f0 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.101+0000 7fcdf9ffb700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 0x7fcdf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.101+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdf4040e90 con 0x7fce04060280 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.102+0000 7fce08ab3700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 0x7fcdf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.102+0000 7fce08ab3700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 0x7fcdf00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fce000096a0 tx=0x7fce00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.106+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdf4014420 con 0x7fce04060280 2024-02-23T00:35:52.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.314+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce04059100 con 0x7fce04060280 2024-02-23T00:35:52.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.315+0000 7fcdf9ffb700 1 -- 172.21.15.118:0/3084010854 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcdf401d980 con 0x7fce04060280 2024-02-23T00:35:52.317 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:52.317 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:52.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 msgr2=0x7fcdf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 0x7fcdf00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fce000096a0 tx=0x7fce00006b40).stop 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 msgr2=0x7fce0410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fcdf40089f0 tx=0x7fcdf400b040).stop 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 shutdown_connections 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fcdf002ff10 0x7fcdf00323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 --2- 172.21.15.118:0/3084010854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce04060280 0x7fce0410cdd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 >> 172.21.15.118:0/3084010854 conn(0x7fce040f0b70 msgr2=0x7fce040f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 shutdown_connections 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:52.317+0000 7fce0b518700 1 -- 172.21.15.118:0/3084010854 wait complete. 2024-02-23T00:35:52.319 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:53.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:52 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:52.315995+0000 mon.smithi090 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.118:0/3084010854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:54.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:53 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:52.363707+0000 mgr.smithi090.pxxmlz (mgr.14162) 384 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:54.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:54.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:35:56.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:55 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:54.364085+0000 mgr.smithi090.pxxmlz (mgr.14162) 385 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.302+0000 7f081bc51700 1 -- 172.21.15.118:0/1799859167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 msgr2=0x7f08140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.302+0000 7f081bc51700 1 --2- 172.21.15.118:0/1799859167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08140f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f0804009fb0 tx=0x7f0804009b30).stop 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.302+0000 7f081bc51700 1 -- 172.21.15.118:0/1799859167 shutdown_connections 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.302+0000 7f081bc51700 1 --2- 172.21.15.118:0/1799859167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08140f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 -- 172.21.15.118:0/1799859167 >> 172.21.15.118:0/1799859167 conn(0x7f08140f0b70 msgr2=0x7f08140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 -- 172.21.15.118:0/1799859167 shutdown_connections 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 -- 172.21.15.118:0/1799859167 wait complete. 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 Processor -- start 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 -- start start 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:56.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f081bc51700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814060240 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f08199ed700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f08199ed700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50462/0 (socket says 172.21.15.118:50462) 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.303+0000 7f08199ed700 1 -- 172.21.15.118:0/3263967174 learned_addr learned my addr 172.21.15.118:0/3263967174 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f08199ed700 1 -- 172.21.15.118:0/3263967174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0804005040 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f08199ed700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0804005010 tx=0x7f080400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0804009320 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0804013380 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814060440 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0804014e50 con 0x7f08140f5410 2024-02-23T00:35:56.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.304+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0814109b90 con 0x7f08140f5410 2024-02-23T00:35:56.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.305+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0804014420 con 0x7f08140f5410 2024-02-23T00:35:56.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.305+0000 7f080a7fc700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 0x7f08000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:56.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.305+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814045ad0 con 0x7f08140f5410 2024-02-23T00:35:56.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.305+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0804040e30 con 0x7f08140f5410 2024-02-23T00:35:56.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.305+0000 7f08191ec700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 0x7f08000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:56.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.306+0000 7f08191ec700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 0x7f08000323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f08100096a0 tx=0x7f0810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:56.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.310+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08040095f0 con 0x7f08140f5410 2024-02-23T00:35:56.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.515+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f081410a3b0 con 0x7f08140f5410 2024-02-23T00:35:56.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.516+0000 7f080a7fc700 1 -- 172.21.15.118:0/3263967174 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f080402b020 con 0x7f08140f5410 2024-02-23T00:35:56.518 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:35:56.518 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:35:56.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.517+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 msgr2=0x7f08000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:56.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 0x7f08000323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f08100096a0 tx=0x7f0810006b40).stop 2024-02-23T00:35:56.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 msgr2=0x7f08141094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0804005010 tx=0x7f080400b040).stop 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 shutdown_connections 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f080002ff10 0x7f08000323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 --2- 172.21.15.118:0/3263967174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08140f5410 0x7f08141094e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 >> 172.21.15.118:0/3263967174 conn(0x7f08140f0b70 msgr2=0x7f08140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 shutdown_connections 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:35:56.518+0000 7f081bc51700 1 -- 172.21.15.118:0/3263967174 wait complete. 2024-02-23T00:35:56.520 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:35:57.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:56 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:35:56.516876+0000 mon.smithi090 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.118:0/3263967174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:35:58.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:57 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:56.364395+0000 mgr.smithi090.pxxmlz (mgr.14162) 386 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:35:58.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:35:58.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:00.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:35:59 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:35:58.364766+0000 mgr.smithi090.pxxmlz (mgr.14162) 387 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:00.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 -- 172.21.15.118:0/326125881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 msgr2=0x7fafec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 --2- 172.21.15.118:0/326125881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec0f66c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fafe8009fb0 tx=0x7fafe8009b30).stop 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 -- 172.21.15.118:0/326125881 shutdown_connections 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 --2- 172.21.15.118:0/326125881 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec0f66c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 -- 172.21.15.118:0/326125881 >> 172.21.15.118:0/326125881 conn(0x7fafec0f1aa0 msgr2=0x7fafec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 -- 172.21.15.118:0/326125881 shutdown_connections 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.130+0000 7faff25c8700 1 -- 172.21.15.118:0/326125881 wait complete. 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff25c8700 1 Processor -- start 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff25c8700 1 -- start start 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff25c8700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff25c8700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec069e20 con 0x7fafec0f62a0 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff15c6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff15c6700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50482/0 (socket says 172.21.15.118:50482) 2024-02-23T00:36:00.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff15c6700 1 -- 172.21.15.118:0/3411699050 learned_addr learned my addr 172.21.15.118:0/3411699050 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.131+0000 7faff15c6700 1 -- 172.21.15.118:0/3411699050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8005040 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7faff15c6700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fafe80089f0 tx=0x7fafe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe8009320 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec069fc0 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafe8013380 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafe8014e20 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.132+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec06a450 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafe80263a0 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec059100 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7fafe1ffb700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 0x7fafd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafe8041240 con 0x7fafec0f62a0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7faff0dc5700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 0x7fafd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:00.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.133+0000 7faff0dc5700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 0x7fafd80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:00.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.137+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafe801cb10 con 0x7fafec0f62a0 2024-02-23T00:36:00.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.343+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafec0fbe30 con 0x7fafec0f62a0 2024-02-23T00:36:00.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.344+0000 7fafe1ffb700 1 -- 172.21.15.118:0/3411699050 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafe801cb10 con 0x7fafec0f62a0 2024-02-23T00:36:00.346 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:00.346 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 msgr2=0x7fafd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 0x7fafd80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fafdc0096a0 tx=0x7fafdc006b40).stop 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 msgr2=0x7fafec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fafe80089f0 tx=0x7fafe800b040).stop 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 shutdown_connections 2024-02-23T00:36:00.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fafd802ff10 0x7fafd80323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:00.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 --2- 172.21.15.118:0/3411699050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafec0f62a0 0x7fafec06b9a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:00.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.345+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 >> 172.21.15.118:0/3411699050 conn(0x7fafec0f1aa0 msgr2=0x7fafec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:00.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.346+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 shutdown_connections 2024-02-23T00:36:00.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:00.346+0000 7faff25c8700 1 -- 172.21.15.118:0/3411699050 wait complete. 2024-02-23T00:36:00.348 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:01.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:00 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:00.345009+0000 mon.smithi090 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.118:0/3411699050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:02.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:01 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:00.365133+0000 mgr.smithi090.pxxmlz (mgr.14162) 388 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:02.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:02.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:04.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:03 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:02.365416+0000 mgr.smithi090.pxxmlz (mgr.14162) 389 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:04.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 -- 172.21.15.118:0/2330952826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 msgr2=0x7f47980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 --2- 172.21.15.118:0/2330952826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47980f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f4788009fb0 tx=0x7f4788009b30).stop 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 -- 172.21.15.118:0/2330952826 shutdown_connections 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 --2- 172.21.15.118:0/2330952826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47980f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 -- 172.21.15.118:0/2330952826 >> 172.21.15.118:0/2330952826 conn(0x7f47980f0b70 msgr2=0x7f47980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 -- 172.21.15.118:0/2330952826 shutdown_connections 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.063+0000 7f47a01f2700 1 -- 172.21.15.118:0/2330952826 wait complete. 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f47a01f2700 1 Processor -- start 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f47a01f2700 1 -- start start 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f47a01f2700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f47a01f2700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4798106cd0 con 0x7f47980f5410 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f479df8e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f479df8e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:50486/0 (socket says 172.21.15.118:50486) 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f479df8e700 1 -- 172.21.15.118:0/3816236062 learned_addr learned my addr 172.21.15.118:0/3816236062 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:04.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f479df8e700 1 -- 172.21.15.118:0/3816236062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788005040 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.064+0000 7f479df8e700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f47880089f0 tx=0x7f478800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4788009320 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4798106ed0 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4788013380 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478801ed80 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4798107360 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4798045ad0 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.065+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f478801e420 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.066+0000 7f478e7fc700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 0x7f4784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.066+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4788041060 con 0x7f47980f5410 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.066+0000 7f479d78d700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 0x7f4784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:04.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.066+0000 7f479d78d700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 0x7f4784032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:04.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.070+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47880095f0 con 0x7f47980f5410 2024-02-23T00:36:04.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.277+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47981081b0 con 0x7f47980f5410 2024-02-23T00:36:04.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.278+0000 7f478e7fc700 1 -- 172.21.15.118:0/3816236062 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f478801cb10 con 0x7f47980f5410 2024-02-23T00:36:04.281 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:04.281 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:04.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 msgr2=0x7f4784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 0x7f4784032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).stop 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 msgr2=0x7f47981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f47880089f0 tx=0x7f478800b040).stop 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 shutdown_connections 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f478402ff60 0x7f4784032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 --2- 172.21.15.118:0/3816236062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47980f5410 0x7f47981067e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 >> 172.21.15.118:0/3816236062 conn(0x7f47980f0b70 msgr2=0x7f47980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 shutdown_connections 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:04.280+0000 7f47a01f2700 1 -- 172.21.15.118:0/3816236062 wait complete. 2024-02-23T00:36:04.283 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:05.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:04 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:04.280134+0000 mon.smithi090 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.118:0/3816236062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:06.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:05 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:04.365703+0000 mgr.smithi090.pxxmlz (mgr.14162) 390 : cluster [DBG] pgmap v374: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:06.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:06.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:08.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:07 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:06.366043+0000 mgr.smithi090.pxxmlz (mgr.14162) 391 : cluster [DBG] pgmap v375: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 -- 172.21.15.118:0/29857253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 msgr2=0x7fc8f00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 --2- 172.21.15.118:0/29857253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f00f9190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fc8e0009fb0 tx=0x7fc8e0009b30).stop 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 -- 172.21.15.118:0/29857253 shutdown_connections 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 --2- 172.21.15.118:0/29857253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f00f9190 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 -- 172.21.15.118:0/29857253 >> 172.21.15.118:0/29857253 conn(0x7fc8f00f0b70 msgr2=0x7fc8f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 -- 172.21.15.118:0/29857253 shutdown_connections 2024-02-23T00:36:08.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 -- 172.21.15.118:0/29857253 wait complete. 2024-02-23T00:36:08.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.382+0000 7fc8f5663700 1 Processor -- start 2024-02-23T00:36:08.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8f5663700 1 -- start start 2024-02-23T00:36:08.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8f5663700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:08.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8f5663700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0106cc0 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35278/0 (socket says 172.21.15.118:35278) 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8eeffd700 1 -- 172.21.15.118:0/3210266901 learned_addr learned my addr 172.21.15.118:0/3210266901 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8eeffd700 1 -- 172.21.15.118:0/3210266901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0005040 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.383+0000 7fc8eeffd700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc8e0007a10 tx=0x7fc8e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e0018070 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f0106ec0 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e0013410 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e001c3c0 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f0107350 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8e001d420 con 0x7fc8f00601e0 2024-02-23T00:36:08.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.384+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f0045ad0 con 0x7fc8f00601e0 2024-02-23T00:36:08.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.385+0000 7fc8d77fe700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:08.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.385+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8e0041570 con 0x7fc8f00601e0 2024-02-23T00:36:08.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.385+0000 7fc8ee7fc700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:08.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.385+0000 7fc8ee7fc700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 0x7fc8d8032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc8e40096a0 tx=0x7fc8e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:08.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.389+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8e001d820 con 0x7fc8f00601e0 2024-02-23T00:36:08.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.592+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8f0108200 con 0x7fc8f00601e0 2024-02-23T00:36:08.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.593+0000 7fc8d77fe700 1 -- 172.21.15.118:0/3210266901 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8e001b370 con 0x7fc8f00601e0 2024-02-23T00:36:08.596 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:08.596 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:08.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 msgr2=0x7fc8d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:08.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 0x7fc8d8032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc8e40096a0 tx=0x7fc8e4006b40).stop 2024-02-23T00:36:08.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 msgr2=0x7fc8f01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc8e0007a10 tx=0x7fc8e0009b30).stop 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 shutdown_connections 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7fc8d802ff60 0x7fc8d8032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 --2- 172.21.15.118:0/3210266901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8f00601e0 0x7fc8f01067d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 >> 172.21.15.118:0/3210266901 conn(0x7fc8f00f0b70 msgr2=0x7fc8f00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 shutdown_connections 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:08.595+0000 7fc8f5663700 1 -- 172.21.15.118:0/3210266901 wait complete. 2024-02-23T00:36:08.598 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:09.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:08 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:08.595160+0000 mon.smithi090 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.118:0/3210266901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:10.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:09 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:08.366419+0000 mgr.smithi090.pxxmlz (mgr.14162) 392 : cluster [DBG] pgmap v376: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:10.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:10.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:12.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:11 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:10.366800+0000 mgr.smithi090.pxxmlz (mgr.14162) 393 : cluster [DBG] pgmap v377: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- 172.21.15.118:0/434339787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 msgr2=0x7f1b740f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 --2- 172.21.15.118:0/434339787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f8f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1b64009fb0 tx=0x7f1b64009b30).stop 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- 172.21.15.118:0/434339787 shutdown_connections 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 --2- 172.21.15.118:0/434339787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f8f20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- 172.21.15.118:0/434339787 >> 172.21.15.118:0/434339787 conn(0x7f1b740f09a0 msgr2=0x7f1b740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- 172.21.15.118:0/434339787 shutdown_connections 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- 172.21.15.118:0/434339787 wait complete. 2024-02-23T00:36:12.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 Processor -- start 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.582+0000 7f1b790ab700 1 -- start start 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b790ab700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b790ab700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b740f4e20 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b73fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b73fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:35290/0 (socket says 172.21.15.118:35290) 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b73fff700 1 -- 172.21.15.118:0/2549162867 learned_addr learned my addr 172.21.15.118:0/2549162867 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b73fff700 1 -- 172.21.15.118:0/2549162867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64005040 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b73fff700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1b64005010 tx=0x7f1b6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b64009320 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b740f4fc0 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b64013380 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b64014e50 con 0x7f1b740601e0 2024-02-23T00:36:12.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.583+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b740f5450 con 0x7f1b740601e0 2024-02-23T00:36:12.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.584+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b64014420 con 0x7f1b740601e0 2024-02-23T00:36:12.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.584+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74059100 con 0x7f1b740601e0 2024-02-23T00:36:12.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.584+0000 7f1b70ff9700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 0x7f1b5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:12.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.584+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b64040e30 con 0x7f1b740601e0 2024-02-23T00:36:12.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.585+0000 7f1b737fe700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 0x7f1b5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:12.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.585+0000 7f1b737fe700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 0x7f1b5c0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1b680096a0 tx=0x7f1b68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:12.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.589+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b640095f0 con 0x7f1b740601e0 2024-02-23T00:36:12.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.796+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b740fbe10 con 0x7f1b740601e0 2024-02-23T00:36:12.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.796+0000 7f1b70ff9700 1 -- 172.21.15.118:0/2549162867 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b6401db70 con 0x7f1b740601e0 2024-02-23T00:36:12.798 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:12.798 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 msgr2=0x7f1b5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 0x7f1b5c0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1b680096a0 tx=0x7f1b68006b40).stop 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 msgr2=0x7f1b740f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1b64005010 tx=0x7f1b6400b040).stop 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 shutdown_connections 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f1b5c02ff10 0x7f1b5c0323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 --2- 172.21.15.118:0/2549162867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b740601e0 0x7f1b740f69a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:12.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.798+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 >> 172.21.15.118:0/2549162867 conn(0x7f1b740f09a0 msgr2=0x7f1b740f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:12.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.799+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 shutdown_connections 2024-02-23T00:36:12.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:12.799+0000 7f1b790ab700 1 -- 172.21.15.118:0/2549162867 wait complete. 2024-02-23T00:36:12.801 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:14.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:12.367176+0000 mgr.smithi090.pxxmlz (mgr.14162) 394 : cluster [DBG] pgmap v378: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:14.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:13 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:12.797376+0000 mon.smithi090 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.118:0/2549162867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:14.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:14.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:16.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:15 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:14.367520+0000 mgr.smithi090.pxxmlz (mgr.14162) 395 : cluster [DBG] pgmap v379: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:16.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 -- 172.21.15.118:0/1003940106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 msgr2=0x7f07680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:16.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 --2- 172.21.15.118:0/1003940106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07680f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f075c009fb0 tx=0x7f075c009b30).stop 2024-02-23T00:36:16.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 -- 172.21.15.118:0/1003940106 shutdown_connections 2024-02-23T00:36:16.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 --2- 172.21.15.118:0/1003940106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07680f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:16.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 -- 172.21.15.118:0/1003940106 >> 172.21.15.118:0/1003940106 conn(0x7f07680f0b70 msgr2=0x7f07680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 -- 172.21.15.118:0/1003940106 shutdown_connections 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.844+0000 7f076cfb5700 1 -- 172.21.15.118:0/1003940106 wait complete. 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076cfb5700 1 Processor -- start 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076cfb5700 1 -- start start 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076cfb5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:16.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076cfb5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0768060240 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076659c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60528/0 (socket says 172.21.15.118:60528) 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076659c700 1 -- 172.21.15.118:0/3608140401 learned_addr learned my addr 172.21.15.118:0/3608140401 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.845+0000 7f076659c700 1 -- 172.21.15.118:0/3608140401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c005040 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f076659c700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f075c005010 tx=0x7f075c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f075c009320 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0768060440 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f075c013380 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0768109b90 con 0x7f07680f5410 2024-02-23T00:36:16.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.846+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f075c014e50 con 0x7f07680f5410 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f075c0263a0 con 0x7f07680f5410 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0768045ad0 con 0x7f07680f5410 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f0756ffd700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 0x7f07500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f075c041220 con 0x7f07680f5410 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f0765d9b700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 0x7f07500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:16.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.847+0000 7f0765d9b700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 0x7f07500323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f07580096a0 tx=0x7f0758006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:16.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:16.852+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f075c012330 con 0x7f07680f5410 2024-02-23T00:36:17.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.057+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f076810a3a0 con 0x7f07680f5410 2024-02-23T00:36:17.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.058+0000 7f0756ffd700 1 -- 172.21.15.118:0/3608140401 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f075c012330 con 0x7f07680f5410 2024-02-23T00:36:17.060 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:17.060 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:17.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 msgr2=0x7f07500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:17.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 0x7f07500323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f07580096a0 tx=0x7f0758006b40).stop 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 msgr2=0x7f07681094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f075c005010 tx=0x7f075c00b040).stop 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 shutdown_connections 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f075002ff10 0x7f07500323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 --2- 172.21.15.118:0/3608140401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07680f5410 0x7f07681094e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 >> 172.21.15.118:0/3608140401 conn(0x7f07680f0b70 msgr2=0x7f07680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 shutdown_connections 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:17.060+0000 7f076cfb5700 1 -- 172.21.15.118:0/3608140401 wait complete. 2024-02-23T00:36:17.062 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:18.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:16.367883+0000 mgr.smithi090.pxxmlz (mgr.14162) 396 : cluster [DBG] pgmap v380: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:18.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:17 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:17.059140+0000 mon.smithi090 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.118:0/3608140401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:18.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:18.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:20.022 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:19 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:18.368244+0000 mgr.smithi090.pxxmlz (mgr.14162) 397 : cluster [DBG] pgmap v381: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 -- 172.21.15.118:0/1070247042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 msgr2=0x7f4ad80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 --2- 172.21.15.118:0/1070247042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad80f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f4acc009fb0 tx=0x7f4acc009b30).stop 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 -- 172.21.15.118:0/1070247042 shutdown_connections 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 --2- 172.21.15.118:0/1070247042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad80f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 -- 172.21.15.118:0/1070247042 >> 172.21.15.118:0/1070247042 conn(0x7f4ad80f0b70 msgr2=0x7f4ad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 -- 172.21.15.118:0/1070247042 shutdown_connections 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.925+0000 7f4ade922700 1 -- 172.21.15.118:0/1070247042 wait complete. 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.926+0000 7f4ade922700 1 Processor -- start 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.926+0000 7f4ade922700 1 -- start start 2024-02-23T00:36:21.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.926+0000 7f4ade922700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.926+0000 7f4ade922700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad8105bd0 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad7fff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:60534/0 (socket says 172.21.15.118:60534) 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad7fff700 1 -- 172.21.15.118:0/1427242171 learned_addr learned my addr 172.21.15.118:0/1427242171 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad7fff700 1 -- 172.21.15.118:0/1427242171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc005040 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad7fff700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4acc005010 tx=0x7f4acc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4acc009320 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad8105dd0 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4acc013380 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.927+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4acc014e50 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.928+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad8106260 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.928+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad8045ad0 con 0x7f4ad80f5410 2024-02-23T00:36:21.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.928+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4acc0095f0 con 0x7f4ad80f5410 2024-02-23T00:36:21.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.928+0000 7f4ad4ff9700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 0x7f4ac00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:21.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.929+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4acc040e70 con 0x7f4ad80f5410 2024-02-23T00:36:21.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.929+0000 7f4ad77fe700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 0x7f4ac00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:21.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.929+0000 7f4ad77fe700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 0x7f4ac00323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4ac80096a0 tx=0x7f4ac8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:21.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:20.933+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4acc014420 con 0x7f4ad80f5410 2024-02-23T00:36:21.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.142+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ad8107110 con 0x7f4ad80f5410 2024-02-23T00:36:21.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.142+0000 7f4ad4ff9700 1 -- 172.21.15.118:0/1427242171 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4acc010230 con 0x7f4ad80f5410 2024-02-23T00:36:21.144 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:21.144 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:21.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 msgr2=0x7f4ac00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 0x7f4ac00323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4ac80096a0 tx=0x7f4ac8006b40).stop 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 msgr2=0x7f4ad81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4acc005010 tx=0x7f4acc00b040).stop 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 shutdown_connections 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f4ac002ff10 0x7f4ac00323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 --2- 172.21.15.118:0/1427242171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ad80f5410 0x7f4ad81056e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 >> 172.21.15.118:0/1427242171 conn(0x7f4ad80f0b70 msgr2=0x7f4ad80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 shutdown_connections 2024-02-23T00:36:21.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:21.144+0000 7f4ade922700 1 -- 172.21.15.118:0/1427242171 wait complete. 2024-02-23T00:36:21.147 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:22.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:20.368549+0000 mgr.smithi090.pxxmlz (mgr.14162) 398 : cluster [DBG] pgmap v382: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:22.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:21 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:21.143171+0000 mon.smithi090 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.118:0/1427242171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:22.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-23T00:36:22.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:36:24.023 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:23 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:22.368951+0000 mgr.smithi090.pxxmlz (mgr.14162) 399 : cluster [DBG] pgmap v383: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:25.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- 172.21.15.118:0/250342223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 msgr2=0x7f53140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 --2- 172.21.15.118:0/250342223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f53140f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f52fc009fb0 tx=0x7f52fc009b30).stop 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- 172.21.15.118:0/250342223 shutdown_connections 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 --2- 172.21.15.118:0/250342223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f53140f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- 172.21.15.118:0/250342223 >> 172.21.15.118:0/250342223 conn(0x7f53140f0b70 msgr2=0x7f53140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- 172.21.15.118:0/250342223 shutdown_connections 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- 172.21.15.118:0/250342223 wait complete. 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 Processor -- start 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.113+0000 7f5319a11700 1 -- start start 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5319a11700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5319a11700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314060240 con 0x7f53140f5410 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:25.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.118:42550/0 (socket says 172.21.15.118:42550) 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5312ffd700 1 -- 172.21.15.118:0/1101407076 learned_addr learned my addr 172.21.15.118:0/1101407076 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5312ffd700 1 -- 172.21.15.118:0/1101407076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc005040 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5312ffd700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f52fc005010 tx=0x7f52fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52fc009320 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314060440 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52fc013380 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52fc014e50 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.114+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5314109bd0 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.115+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52fc0095f0 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.115+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5314045ad0 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.115+0000 7f530b7fe700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 0x7f53000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.115+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52fc040fa0 con 0x7f53140f5410 2024-02-23T00:36:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.116+0000 7f53127fc700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 0x7f53000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.116+0000 7f53127fc700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 0x7f53000323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f53040096a0 tx=0x7f5304006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.120+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52fc014420 con 0x7f53140f5410 2024-02-23T00:36:25.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.327+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f531410a3f0 con 0x7f53140f5410 2024-02-23T00:36:25.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.328+0000 7f530b7fe700 1 -- 172.21.15.118:0/1101407076 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52fc010230 con 0x7f53140f5410 2024-02-23T00:36:25.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-02-23T00:36:25.330 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"898d23fe-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:22:18.065834Z","created":"2024-02-23T00:22:18.065834Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:36:25.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 msgr2=0x7f53000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:25.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 0x7f53000323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f53040096a0 tx=0x7f5304006b40).stop 2024-02-23T00:36:25.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 msgr2=0x7f5314109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:25.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f52fc005010 tx=0x7f52fc00b040).stop 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 shutdown_connections 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:6800/1747781840,v1:172.21.15.90:6801/1747781840] conn(0x7f530002ff10 0x7f53000323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 --2- 172.21.15.118:0/1101407076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53140f5410 0x7f5314109520 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 >> 172.21.15.118:0/1101407076 conn(0x7f53140f0b70 msgr2=0x7f53140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 shutdown_connections 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-02-23T00:36:25.330+0000 7f5319a11700 1 -- 172.21.15.118:0/1101407076 wait complete. 2024-02-23T00:36:25.332 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-02-23T00:36:25.922 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-23T00:36:25.924 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-23T00:36:25.924 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:36:25.948 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:36:25.966 INFO:tasks.cephadm:Stopping all daemons... 2024-02-23T00:36:25.966 INFO:tasks.cephadm.mon.smithi090:Stopping mon.smithi090... 2024-02-23T00:36:25.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090 2024-02-23T00:36:26.019 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: cluster 2024-02-23T00:36:24.369202+0000 mgr.smithi090.pxxmlz (mgr.14162) 400 : cluster [DBG] pgmap v384: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:36:26.019 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:25.256453+0000 mon.smithi090 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:36:26.019 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:25.262287+0000 mon.smithi090 (mon.0) 342 : audit [INF] from='mgr.14162 172.21.15.90:0/1819020914' entity='mgr.smithi090.pxxmlz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.pxxmlz/trash_purge_schedule"}]: dispatch 2024-02-23T00:36:26.019 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:25 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: audit 2024-02-23T00:36:25.328974+0000 mon.smithi090 (mon.0) 343 : audit [DBG] from='client.? 172.21.15.118:0/1101407076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:36:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:26 smithi090 systemd[1]: Stopping Ceph mon.smithi090 for 898d23fe-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:36:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:36:26.161+0000 7fa3cf3e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi090 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:36:26.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:26 smithi090 ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090[38180]: debug 2024-02-23T00:36:26.161+0000 7fa3cf3e0700 -1 mon.smithi090@0(leader) e1 *** Got Signal Terminated *** 2024-02-23T00:36:26.773 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Feb 23 00:36:26 smithi090 podman[44167]: ceph-898d23fe-d1e1-11ee-95c0-87774f69a715-mon.smithi090 2024-02-23T00:36:26.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi090.service' 2024-02-23T00:36:26.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:36:26.908 INFO:tasks.cephadm.mon.smithi090:Stopped mon.smithi090 2024-02-23T00:36:26.908 INFO:tasks.cephadm.mon.smithi118:Stopping mon.smithi118... 2024-02-23T00:36:26.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118 2024-02-23T00:36:26.945 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118.service: Unit ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118.service not loaded. 2024-02-23T00:36:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-23T00:36:26.947 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi118" Traceback (most recent call last): File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 5: 'sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118' 2024-02-23T00:36:26.948 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-23T00:36:26.950 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/ubuntu@smithi090.front.sepia.ceph.com/crash 2024-02-23T00:36:26.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/crash -- . 2024-02-23T00:36:27.016 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/ubuntu@smithi118.front.sepia.ceph.com/crash 2024-02-23T00:36:27.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/crash -- . 2024-02-23T00:36:27.082 INFO:teuthology.orchestra.run.smithi118.stderr:tar: /var/lib/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/crash: Cannot open: No such file or directory 2024-02-23T00:36:27.083 INFO:teuthology.orchestra.run.smithi118.stderr:tar: Error is not recoverable: exiting now 2024-02-23T00:36:27.083 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-23T00:36:27.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CEPHADM_STRAY_DAEMON\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v FS_ | egrep -v '\(CEPHADM_' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v acting | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'is down' | egrep -v 'osds down' | head -n 1 2024-02-23T00:36:27.123 INFO:tasks.cephadm:Compressing logs... 2024-02-23T00:36:27.123 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-23T00:36:27.166 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-23T00:36:27.192 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-23T00:36:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:: No such file or directory 2024-02-23T00:36:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log 2024-02-23T00:36:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.audit.log 2024-02-23T00:36:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log 2024-02-23T00:36:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log: /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.log 2024-02-23T00:36:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log: 88.5% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log.gz 2024-02-23T00:36:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mgr.smithi090.pxxmlz.log 2024-02-23T00:36:27.194 INFO:teuthology.orchestra.run.smithi090.stderr: 90.3% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log.gz 2024-02-23T00:36:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mon.smithi090.log 2024-02-23T00:36:27.195 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.audit.log.gz 2024-02-23T00:36:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.log: /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mgr.smithi090.pxxmlz.log: 88.7% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph.log.gz 2024-02-23T00:36:27.195 INFO:teuthology.orchestra.run.smithi118.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-23T00:36:27.196 INFO:teuthology.orchestra.run.smithi118.stderr:: No such file or directory 2024-02-23T00:36:27.196 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log 2024-02-23T00:36:27.196 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-volume.log.gz 2024-02-23T00:36:27.196 INFO:teuthology.orchestra.run.smithi118.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-23T00:36:27.198 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-02-23T00:36:27.198 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m0.014s 2024-02-23T00:36:27.198 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m0.013s 2024-02-23T00:36:27.198 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.010s 2024-02-23T00:36:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mon.smithi090.log: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-23T00:36:27.295 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mgr.smithi090.pxxmlz.log.gz 2024-02-23T00:36:27.370 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/898d23fe-d1e1-11ee-95c0-87774f69a715/ceph-mon.smithi090.log.gz 2024-02-23T00:36:27.372 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-02-23T00:36:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m0.190s 2024-02-23T00:36:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m0.292s 2024-02-23T00:36:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.017s 2024-02-23T00:36:27.373 INFO:tasks.cephadm:Archiving logs... 2024-02-23T00:36:27.373 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/ubuntu@smithi090.front.sepia.ceph.com/log 2024-02-23T00:36:27.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-23T00:36:27.512 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/ubuntu@smithi118.front.sepia.ceph.com/log 2024-02-23T00:36:27.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-23T00:36:27.547 INFO:tasks.cephadm:Removing cluster... 2024-02-23T00:36:27.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 --force 2024-02-23T00:38:31.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 898d23fe-d1e1-11ee-95c0-87774f69a715 --force 2024-02-23T00:38:32.163 INFO:tasks.cephadm:Removing cephadm ... 2024-02-23T00:38:32.164 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-23T00:38:32.183 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-23T00:38:32.200 INFO:tasks.cephadm:Teardown complete 2024-02-23T00:38:32.201 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 5: 'sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118' 2024-02-23T00:38:32.438 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e5eec5e3ad34448a85a8b420f1c5bb4b Traceback (most recent call last): File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 5: 'sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118' 2024-02-23T00:38:32.443 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-23T00:38:32.452 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-23T00:38:32.459 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-23T00:38:32.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-23T00:38:32.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-23T00:38:32.551 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-23T00:38:32.561 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-23T00:38:32.561 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:38:32.564 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:38:32.579 INFO:teuthology.orchestra.run.smithi090.stderr:bash: ntpq: command not found 2024-02-23T00:38:32.581 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2024-02-23T00:38:32.583 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:38:32.583 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-02-23T00:38:32.584 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +2692us[+2692us] +/- 114ms 2024-02-23T00:38:32.584 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -476us[ -465us] +/- 66ms 2024-02-23T00:38:32.584 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -1317us[-1307us] +/- 89ms 2024-02-23T00:38:32.584 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +2695us[+2695us] +/- 114ms 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 7 377 51 -498us[ -494us] +/- 66ms 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -1338us[-1334us] +/- 89ms 2024-02-23T00:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:38:32.587 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-23T00:38:32.596 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-23T00:38:32.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-23T00:38:32.607 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-23T00:38:32.689 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-23T00:38:32.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:38:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:38:32.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:38:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:38:32.813 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-23T00:38:32.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-23T00:38:32.832 INFO:teuthology.task.internal:Duration was 2090.355485 seconds 2024-02-23T00:38:32.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-23T00:38:32.841 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-23T00:38:32.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-23T00:38:32.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:38:32.899 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:38:33.285 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-23T00:38:33.285 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-02-23T00:38:33.285 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-23T00:38:33.308 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-02-23T00:38:33.308 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-23T00:38:33.331 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-23T00:38:33.332 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-23T00:38:33.351 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-23T00:38:33.421 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-23T00:38:33.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-23T00:38:33.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-23T00:38:33.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-23T00:38:33.527 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-23T00:38:33.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-23T00:38:33.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-23T00:38:33.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-23T00:38:33.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:38:33.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:38:33.621 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-02-23T00:38:33.648 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-02-23T00:38:33.702 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:38:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:38:33.719 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:38:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:38:33.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-23T00:38:33.771 INFO:teuthology.task.internal:Transferring archived files... 2024-02-23T00:38:33.772 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/smithi090 2024-02-23T00:38:33.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-23T00:38:33.813 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395/remote/smithi118 2024-02-23T00:38:33.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-23T00:38:33.852 INFO:teuthology.task.internal:Removing archive directory... 2024-02-23T00:38:33.853 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-23T00:38:33.855 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-23T00:38:33.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-23T00:38:33.915 INFO:teuthology.task.internal:Not uploading archives. 2024-02-23T00:38:33.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-23T00:38:33.943 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-23T00:38:33.944 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-23T00:38:33.946 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-23T00:38:33.961 INFO:teuthology.orchestra.run.smithi090.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 00:38 /home/ubuntu/cephtest 2024-02-23T00:38:33.963 INFO:teuthology.orchestra.run.smithi118.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 00:38 /home/ubuntu/cephtest 2024-02-23T00:38:33.965 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-23T00:38:33.975 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-23T00:38:33.995 INFO:teuthology.nuke:Checking targets against current locks 2024-02-23T00:38:34.022 DEBUG:teuthology.nuke:shortname: smithi090 2024-02-23T00:38:34.022 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-23T00:38:34.049 DEBUG:teuthology.nuke:shortname: smithi118 2024-02-23T00:38:34.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-23T00:38:34.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:54:10.360843', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBAw66Tluu/5EKzyB/erNNt8aJIbaZk+GzOE3LkTuiVuu3criI50enx46TASsXmmP3ZNqFnmQbRlfJqHlCr+Jfs='} 2024-02-23T00:38:34.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571395', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:54:10.362575', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpYTI8SAlR2H/0ngjM5LP1E9T4AWk2Q31Wy3Sdi4GjlZr53ndDadIrd5zUNflUMiaEgrf1uQxihOO+pcT4DKwE='} 2024-02-23T00:38:34.108 INFO:teuthology.orchestra.console.smithi090:Power off 2024-02-23T00:38:34.108 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-23T00:38:34.137 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-02-23T00:38:34.137 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:34.156 INFO:teuthology.orchestra.console.smithi118:Power off 2024-02-23T00:38:34.156 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-23T00:38:34.167 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-23T00:38:34.181 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-02-23T00:38:34.182 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:34.207 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-02-23T00:38:38.169 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:38.208 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:38.279 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-02-23T00:38:38.319 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-02-23T00:38:42.284 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:42.320 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:38:42.396 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-02-23T00:38:42.431 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-02-23T00:38:42.497 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-02-23T00:38:42.532 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-02-23T00:38:42.637 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 2090.355485200882 failure_reason: 'Command failed on smithi118 with status 5: ''sudo systemctl stop ceph-898d23fe-d1e1-11ee-95c0-87774f69a715@mon.smithi118''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e5eec5e3ad34448a85a8b420f1c5bb4b status: fail success: false 2024-02-23T00:38:42.637 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-23T00:38:42.719 INFO:teuthology.run:FAIL